AES Software Engineer Interview Questions & Guide 2026

The 60-second brief

A Software Engineer at AES plays a pivotal role in the digital transformation of global energy infrastructure. You are not just writing code; you are building the systems that monitor, optimize, and secure the power grids and renewable energy assets that supply electricity to millions. Your work directly impacts how energy is generated, transmitted, and distributed, making this a high-stakes environment where precision and reliability are paramount.

For a design discussion, a catalogue of architectures is worth less than the ability to turn a vague requirement into a data model and an API contract. A box diagram with no schema under it collapses at the first follow-up question.

Focus areas

• Choose indexes from the query's access path
• Make every write idempotent under retry
• Paginate large result sets with keyset cursors

Overview

The role involves bridging the gap between complex engineering challenges and software solutions. Whether you are analyzing grid capacity, improving transmission system efficiency, or developing tools for renewable energy projects, you will operate at the intersection of software development and industrial operations. You will often collaborate with cross-functional teams—including electrical engineers, grid operators, and project managers—to deliver solutions that are both technically robust and operationally viable.

This is a position for engineers who thrive on solving "real-world" problems. You will encounter unique challenges related to large-scale data, regulatory compliance, and physical infrastructure. Success in this role requires a blend of deep technical proficiency and an appreciation for the critical nature of the energy sector, where the software you build has a tangible, physical impact on the world.

The AES interview process

AES candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screen

The title covers product work, platform work, infrastructure, mobile and frontend, and those are different jobs with different loops behind them. A screening call is the cheapest place to find out which one the seat is, and asking reads as experienced rather than fussy. The questions that separate them: what the team is on call for, what the last three projects were, and whether any round happens inside an existing repository instead of a blank file. Then say which of that you have done and which you have not. Claiming the whole posting is the fastest way to be found out one round later.

What to demonstrate

• Whether you can locate your experience inside one flavour of the role honestly instead of claiming the entire requirements list
• Whether you name what you have not done, which an experienced screener reads as a level signal and can plan the loop around
• Whether what you want next matches what the seat is: someone who wants greenfield work landing on a team that mostly operates an existing system is a hire that leaves within the year

How to prepare

• Mark every line of the posting as done, adjacent or new, and write one sentence for each adjacent line naming the closest thing you actually built
• Split your last two years into rough percentages across feature work, operating and debugging live systems, and design or review, so a question about scope gets numbers rather than adjectives
• Bring three questions that discriminate between seats: what the team is paged for, how much of the work is changing existing code versus standing up something new, and what shipped in the last quarter

Technical Interviews

What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.

What to demonstrate

• Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
• Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
• Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly

How to prepare

• Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
• Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
• Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live

Behavioral Interviews

What you say here is written down by each interviewer and compared afterwards, so the unit of evaluation is a claim someone else could check, not a well-told narrative. Two things make a story checkable: detail only a participant would hold, and a clean line around which part was yours. Vague ownership is the usual failure and it is usually accidental, because engineers say we about the team's work and we about their own, so the thing they personally built disappears into the plural. Name the part you wrote, and name who did the rest.

What to demonstrate

• Whether your details are ones a participant would hold and an observer would not: the constraint that ruled out the obvious approach, the first attempt that failed, the person who objected and on what grounds
• Whether ownership survives a direct question, since a follow-up to we decided is routinely who decided, and an answer that stays plural at that point is read as the work belonging to someone else
• Whether the numbers you quote are ones you would say identically to a former colleague with the dashboard open

How to prepare

• Go through each story replacing every we with either I or a named role (the on-call engineer, the reviewer, the other team) and check the story still holds together. Wherever it stops making sense you have found a part you cannot actually speak to
• Open the artefacts for two of your stories, the pull request, the design doc, the incident notes, and read them for dates and figures you have been rounding in the retelling. Correct your version to match
• For each story write the single sentence you would least want repeated to a former teammate, then either make it accurate or take it out

Panel Interviews

Nobody in the room with you decides this. Interviewers typically write their rounds up separately, often before seeing anyone else's, and the outcome is settled later from those write-ups. A split panel gets resolved by whichever note carries specific evidence, so what you want out of each room is one concrete thing that person could write down: a bug you caught yourself, a trade-off you named, a decision you owned. The rest is arithmetic. The project you describe in a behavioural conversation is often the same system you sketched an hour earlier, and the two accounts have to agree.

What to demonstrate

• Whether the scale, team size and timeline you attach to a project hold steady when that project resurfaces in a different round
• Whether each interviewer leaves with a specific thing to cite rather than a general impression of competence
• Whether a trade-off you defended in one round survives a challenge in another, instead of being quietly swapped for the answer the new interviewer seemed to want
• Whether a question you have already answered earlier in the day gets the same answer at the same depth, without visible impatience

How to prepare

• Write a one-page sheet per project fixing the figures you will quote — request volume, data size, team size, elapsed time, what broke — and say them aloud from the sheet until they come out identical every time
• For each round on the schedule, decide in advance the one sentence you want in that person's notes, then check in a mock that you said it outright instead of leaving it to be inferred
• Have someone ask you the same project question twice, an hour apart, and diff the two answers for numbers that moved or a trade-off that reversed

Remote Assessments

Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.

What to demonstrate

• Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
• Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
• Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
• Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply

How to prepare

• For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
• For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
• Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last

Where candidates lose points

Letting a slow dependency consume unbounded concurrency

The failure that takes a service down is usually not an error but a delay. A dependency answering in thirty seconds instead of fifty milliseconds holds each request's worker or connection six hundred times longer, and since required concurrency is arrival rate times latency, a fleet sized for sixty in-flight requests now needs thirty-six thousand to sustain the same rate - so it queues, and requests whose clients have already abandoned them still occupy resources. Retries make it precisely worse: a policy of three attempts triples the load on a dependency at the exact moment it is least able to serve, which is how one slow dependency becomes an outage of everything sharing that pool. Containment is four specific things - a timeout on every outbound call shorter than the caller's remaining budget, a bounded pool per dependency so one cannot starve the others, backoff with full jitter rather than a fixed delay so retries do not resynchronise, and a circuit that stops sending once the failure rate makes an attempt pointless.

Shipping a migration and the code that depends on it as a single change

During any rolling deploy, and for as long as a rollback remains possible, old and new code execute against the same schema at the same time. A migration that drops or renames a column breaks every instance that has not restarted yet, and code that requires a column the migration has not applied breaks every instance that restarted early. The discipline is expand then contract: add the new column nullable, write both shapes, backfill in batches, move reads across once the backfill is verified, and only then stop writing the old shape and drop it - four deploys, usually spread over days. It feels disproportionate until the first rollback, at which point it is the only reason the previous version still runs.

Saying 'eventually consistent' without naming the anomaly a user would see

Describe the concrete symptom you are choosing to accept: the author reloads and their own comment is missing for two seconds, or two devices show different balances for a minute. The class of consistency model is a technical label; the tolerable anomaly is the actual product decision.

Comparing floating-point values for equality, or holding money in them

Binary floating point cannot represent 0.1 exactly, so repeated addition drifts and an equality check fails on values that are mathematically equal. Store currency as integer minor units or a decimal type, and compare floats against a tolerance you chose for a stated reason.

Questions & practice

How good are you at analyzing data?

Medium · data structures and algorithms

Approach

1. Choose the data structure from the access pattern, not from familiarity.
2. State the target complexity and say which constraint rules the naive version out.
3. Restate the input: its shape, its size, and what is guaranteed about it.

Follow-up

• How does this change if the input no longer fits in memory?
• What is the worst case, and how likely is it on real data?

How would your current experience help you succeed in this role?

Medium · data structures and algorithms

Approach

1. Choose the data structure from the access pattern, not from familiarity.
2. Restate the input: its shape, its size, and what is guaranteed about it.
3. Name the brute-force solution and its complexity before improving on it.

Follow-up

• What is the worst case, and how likely is it on real data?
• Which test case would catch an off-by-one here?

Tell us about your relevant experiences in the field and how you overcame common challenges.

Medium · data structures and algorithms

Approach

1. Restate the input: its shape, its size, and what is guaranteed about it.
2. State the target complexity and say which constraint rules the naive version out.
3. Name the brute-force solution and its complexity before improving on it.

Follow-up

• What is the worst case, and how likely is it on real data?
• Which test case would catch an off-by-one here?

Diff a projection against the primary without per-row point reads

Hard · reconciliation · range hashing · throttling

The listing projection has drifted and some rows show a stale version. The primary holds 40,000,000 resource rows across 12,000 tenants while serving 1,200 writes and 14,000 reads per second. The obvious repair, reading each resource row and comparing its version against the projection, is correct and would eventually finish. Explain precisely why it is unacceptable here, then give a diff that finds the differing rows, state its complexity, and make it safe to run against a live primary. Replication lag is usually under 100 ms and is not bounded.

Approach

1. Quantify the naive cost rather than calling it slow: 40,000,000 point reads at even 0.5 ms each is over five hours serialised, and the only lever is concurrency, which is exactly what you cannot spend. The primary's pool is sized for the write path, and 40,000,000 random reads evict the buffer cache that sustains the 85 percent cache hit rate, so the audit degrades the system it is auditing.
2. Replace random access with one ordered pass per side. Both sides can be read in (tenant_id, resource_id) order, which is a sequential scan on each and a merge join in O(n) time and O(1) memory. For a dense diff that is the whole answer, and it reads the primary once instead of 40,000,000 times.
3. For the expected sparse case, compare range hashes instead of rows: partition the key space, compute per range an order-independent aggregate over hash(resource_id, version), compare aggregates, and descend only into ranges that differ. With d differing rows and branching factor B, at most d ranges mismatch per level, so the drill-down examines O(d log_B(n/d)) ranges and reads full rows only in mismatching leaves.
4. Aggregate with a sum modulo 2^64 or a multiset hash, never XOR. XOR is order-independent but self-cancelling, so two rows wrong in the same way, or a row duplicated on one side, leave the range aggregate matching and the range is declared clean.
5. Pin the comparison to a point in time or it reports lag as drift: consider only rows whose updated_at is older than now minus a lag margin, and re-check each candidate mismatch individually before repairing. At 1,200 writes per second a diff without this reports thousands of false positives, and an unattended repairer would then overwrite live rows with stale values.
6. Make the run resumable and throttled: batch by range key, persist the last completed range, and watch a signal such as replica lag or primary CPU, pausing rather than pressing on. A reconciliation that cannot be stopped and resumed gets killed halfway and restarted from zero, which is how a repair becomes an incident.

Worked solution (35 min)

1. Compute the naive cost explicitly at 40,000,000 reads and 0.5 ms each, then at 100 concurrent, and state what those connections do to a pool already carrying 1,200 writes per second.
2. Write the merge-join version over (tenant_id, resource_id) and state its memory.
3. Define the range aggregate: the range key, the per-row hash input, and the combining function, with one sentence excluding XOR.
4. Work an example with 40,000,000 rows, branching factor 256 and 5 differing rows, and count the ranges examined.
5. Add the watermark filter and the resume point, and name the throttle signal the loop watches.

Expected result: A rejection of per-row point reads backed by the time cost and the cache-eviction argument, a single ordered merge join as the dense-case answer at O(n) time and O(1) memory, a range-hash drill-down examining O(d log_B(n/d)) ranges using a sum or multiset hash rather than XOR, a watermark excluding recently written rows, and a resumable throttled run loop.

Verify it yourself

• Injecting one known stale row makes exactly one leaf range mismatch and the drill-down reaches it.
• Duplicating a row on one side is detected, which is precisely the case XOR would miss.
• Running the diff twice against an idle system reports zero mismatches both times.

Follow-up

• The diff reports 900 stale rows. How do you decide between patching those rows and rebuilding the projection from resource_revision?
• Same job, but the projection lives in a search index that cannot be scanned in key order. What changes?
• How would you run this continuously at low cost instead of only as incident response?

Stop tag and share joins from fanning out a page

Medium · join fan-out · semi-joins · many-to-many

resource_tag is (resource_id, tag_id) with PK (resource_id, tag_id); resource_share is (resource_id, shared_with_user_id, permission). The tagged-and-shared listing inner-joins resource to both, filters tenant_id, tag_id = ANY($2) and shared_with_user_id = $3, orders by updated_at DESC and takes 50. Pages come back with fewer than 50 distinct resources and the total in the header is far too high. Explain the row multiplication, rewrite both the page query and the count query so each is correct, and name the index each one needs. PostgreSQL 16.

Approach

1. Do the arithmetic against the predicates that are actually there. An inner join emits one row per matching child row, and both joins are filtered: tag_id = ANY($2) admits only the requested tags, shared_with_user_id = $3 admits one user's share rows. So a resource holding three of the requested tags and shared with $3 once yields three rows, not one — the multiplier is its count of matching tags times its share rows for that single user, and that second factor is 1 unless the table admits duplicate (resource_id, shared_with_user_id) pairs. LIMIT 50 then limits rows rather than resources, and COUNT(*) counts pairs — the header is the product, not the population.
2. Reject DISTINCT as the fix. It deduplicates after the product has been built, so the planner must materialise and sort the fanned-out set before the LIMIT can apply, and it leaves any SUM or AVG in the same select list wrong.
3. Rewrite both filters as semi-joins, keeping resource as the only row source: AND EXISTS (SELECT 1 FROM resource_tag rt WHERE rt.resource_id = r.resource_id AND rt.tag_id = ANY($2)) and the same shape against resource_share. A semi-join stops at the first match per resource and preserves the driving index order, so ORDER BY updated_at DESC, resource_id DESC LIMIT 50 still stops after 50 rows.
4. Count with the same predicates and no join at all: SELECT count(*) FROM resource r WHERE r.tenant_id = $1 AND r.status = 'active' AND EXISTS (...) AND EXISTS (...). Nothing multiplies a resource, so the number is the population.
5. Attach the tags for display after the page has been cut — LEFT JOIN LATERAL (SELECT array_agg(rt.tag_id) FROM resource_tag rt WHERE rt.resource_id = p.resource_id) ON TRUE over the 50 returned rows. Aggregate over the page, never over the tenant.
6. Index both directions and say which query each serves: PK (resource_id, tag_id) serves the lateral lookup, (tag_id, resource_id) serves the EXISTS probe by tag, and resource_share needs (shared_with_user_id, resource_id) for the same reason. An index covering one direction only leaves the other as a scan.

Worked solution (30 min)

1. Build a tenant where each resource carries 0-5 tags from a 20-tag vocabulary and is shared with 0-4 distinct users, then bind $2 to three tags and $3 to a user holding shares on about half the resources. Run the joined query and compare its row count to the distinct resource count on page one.
2. Run COUNT(*) on the joined shape and on the EXISTS shape and compare both to a ground truth computed from distinct ids; then give $3 a second permission row on 10% of resources and record which of the two counts moves.
3. EXPLAIN both page queries and compare rows-read plus the presence of a Sort or HashAggregate node above the join.
4. Add (tag_id, resource_id), re-run the EXISTS probe, and record the plan change on the inner side.

Expected result: The EXISTS page returns exactly 50 distinct resources and reads about 50 driving rows with no aggregate above the join. The EXISTS count matches the ground truth. The joined COUNT(*) is larger by roughly the average number of matching tags per qualifying resource — the tags it holds from $2, not its total tag count — times its share rows for $3, which is 1 until the duplicate permission rows exist and rises with them afterwards. The EXISTS count is unchanged by those duplicates.

Verify it yourself

• The page's resource_id list has no duplicates and is 50 long whenever at least 50 rows match.
• count(*) from the rewrite equals count(DISTINCT r.resource_id) over the joined shape.
• EXPLAIN shows a semi-join node on the inner side rather than a full join followed by a unique or aggregate step.

Follow-up

• The filter changes from 'any of these tags' to 'all of these tags'. Rewrite it and state what it costs relative to the ANY form.
• A resource can be shared with the same user twice under different permissions. Does your count change, and should it?
• Where does the correct total come from when the tenant holds 4M resources and the header must not cost 200 ms?

Write the update path that detects a concurrent edit

Medium · optimistic concurrency · transactional outbox · audit history

resource carries version INT NOT NULL DEFAULT 1. resource_revision holds revision_id, resource_id, version, actor_user_id, change_kind, patch JSONB, request_id, created_at with UNIQUE (resource_id, version). outbox_event holds aggregate_type, aggregate_id, aggregate_version, event_type, payload, status. A PUT carries the version the client read. Write the exact statements for the single transaction that applies the edit, records the revision and enqueues 'resource.updated', and give the handler's branch on zero affected rows. Then say what PostgreSQL 16 does under READ COMMITTED when two of these updates hit one row at once.

Approach

1. One transaction, three writes, no network call inside it: UPDATE resource SET title = $3, version = version + 1, updated_at = now() WHERE resource_id = $1 AND tenant_id = $4 AND version = $2; then INSERT the resource_revision row at version $2 + 1; then INSERT the outbox_event row at the same aggregate_version. The event goes to a table rather than a broker because no transaction spans both.
2. Branch on the affected-row count before doing anything else. Zero has three causes — stale version, wrong tenant, row gone — so re-read once and map to 409 carrying the current version, or 404 for an id outside the caller's tenant, which also stops the endpoint confirming that another tenant's id exists.
3. State the engine behaviour instead of assuming it. Under READ COMMITTED the second UPDATE blocks on the row lock, and when the first commits PostgreSQL re-evaluates the WHERE clause against the newly committed row, so the version predicate now fails and the statement reports zero rows. Under REPEATABLE READ the identical collision raises SQLSTATE 40001 instead, so the handler must fold both shapes into one conflict response.
4. Keep UNIQUE (resource_id, version) even though the predicate already serialises writers. It is what makes a lost update unwritable if any other path ever reaches the revision table, and it converts a logic bug into 23505 rather than into a silently missing history row.
5. Refuse to auto-retry the whole PUT. A retry re-reads the winner's state and reapplies an intent formed against data that no longer exists — the silent overwrite the version token was added to detect. Return the conflict; merge field-wise only if the patches are provably disjoint.
6. Note that now() is the transaction timestamp in PostgreSQL, so resource.updated_at, the revision's created_at and the outbox row share one instant, which is what later makes reconciliation between the three tables unambiguous.

Follow-up

• A client sends the version it read ten minutes ago and the resource has moved three versions. What is in your 409 so it can resolve the conflict without a full re-fetch?
• Two editors, two disjoint fields, no overlap. Does your answer still refuse the second write, and should it?
• Every write now touches a second hot table. How do you keep the outbox insert and its partial index from becoming the write bottleneck at 1.2k writes/second?

What do you know about the interconnection process?

Medium

Approach

1. Clarify what is being asked and what a complete answer contains.
2. State your assumptions explicitly before working the problem.
3. Say what you would check first and why it is the highest-information step.

Follow-up

• How would you know your answer was wrong?
• What assumption would you test first?

How do you analyze capacity on the grid and what is your strategy around it?

Medium

Approach

1. Clarify what is being asked and what a complete answer contains.
2. Work from the requirement backwards to the design.
3. State your assumptions explicitly before working the problem.

Follow-up

• How would you know your answer was wrong?
• What assumption would you test first?

Make resource creation safe for a client that retried a timeout

Medium · idempotency · retries · race conditions

POST /v1/resources creates a resource row and, in the same transaction, a resource_revision and an outbox_event. A partner's HTTP client times out at 2 seconds and retries twice; under load the first request is still executing when the retry arrives, and duplicate resources are appearing. Using idempotency_key (tenant_id, idempotency_key, request_fingerprint, state, response_status, response_body, resource_id, locked_until, expires_at), specify the whole contract: first call, concurrent retry, retry after completion, a key reused with a different body, and what the client does with each response.

Approach

1. Name why the plain REST shape fails here. POST is not idempotent, a timeout leaves the client unable to distinguish a lost request from a lost response, so it must retry - and the retry is a second create. Checking whether the key exists and then inserting it is check-then-act: under exactly the concurrency that caused the timeout, both attempts look, both see nothing, both proceed.
2. Let the unique constraint arbitrate instead. INSERT the key row with state 'in_flight' and locked_until set to now plus the request deadline, and commit that reservation on its own before any work is attempted - an uncommitted row is invisible to the concurrent retry, which is the entire point of writing it first. One caller wins the insert and proceeds; the other takes the unique violation and reads the row rather than falling through to the work.
3. Branch the loser on what it found. 'succeeded' replays response_status and response_body verbatim, with a header marking it a replay. 'in_flight' with locked_until in the future returns 409 and a Retry-After - do not park a request thread waiting on another request, that is how a concurrency bound is consumed by retries. 'in_flight' past locked_until is reclaimable by a conditional UPDATE predicated on the old locked_until, so exactly one reclaimer wins. A fingerprint mismatch is 422 and never the stored response: a different body under the same key is a client defect, and serving the old response hides it forever.
4. Commit the effect and the key's completion together - resource insert, revision row, outbox row, and the UPDATE of the reserved key to 'succeeded' with the stored response, in one transaction. That makes a crash binary: either all of it is durable and the key reads 'succeeded', or the transaction rolls back and the only durable trace is the reservation row, still 'in_flight', with no resource behind it. The reclaim branch finishes that case - once locked_until passes, one retry takes the row by conditional UPDATE and runs the work for the first time. Splitting the two commits is what creates the window with no recovery: the effect durable, the key still 'in_flight', and a reclaimer that cannot distinguish that from work never started, so it creates a second resource.
5. State the lifetime explicitly. Keys expire - 24 hours is a common choice - and are swept, so the client's total retry window must be shorter than the TTL, or a late retry re-executes with no record that it is a duplicate. The key is scoped by tenant so two tenants choosing the same string do not collide, and the client generates one key per logical attempt, not per network send.
6. Write the client rules: on timeout, retry the same key with capped backoff; on 409, wait Retry-After and retry the same key; on 422, stop and fix the caller; on 2xx, treat the body as authoritative whether it was executed or replayed.

Worked solution (30 min)

1. Write the two-request interleaving that defeats read-then-insert, with the line each request is executing at each step.
2. Write the insert-first sequence and the four branches taken on unique violation, as a table of state, locked_until and response.
3. Put the completion UPDATE inside the same transaction as the resource, revision and outbox writes, then state what a crash before that commit leaves durable and which branch cleans it up.
4. Define the key's scope, lifetime and generation rule, then bound the client's retry budget by the TTL.
5. Fire two concurrent requests carrying one key and count rows in resource and resource_revision.

Expected result: One resource row, one revision row and one outbox row; the second caller either replays the first response or receives 409 with Retry-After; a different body under the same key is rejected 422; no request thread blocks waiting on another request.

Verify it yourself

• A hundred runs of two simultaneous same-key requests produce exactly one resource and one revision every time.
• Killing the process mid-work leaves zero resource rows and the reservation still 'in_flight'; after locked_until passes a retry reclaims the key and executes, so the crash and the retry together produce exactly one resource row.
• The documented client retry budget is strictly shorter than expires_at minus created_at.

Follow-up

• Response bodies are pruned at 24 hours but resource_id is kept. What do you return to a replay that arrives after pruning?
• The endpoint now also sends a welcome email. Which part of this contract prevents a second send, and which part cannot?
• What exactly breaks if the key is scoped globally rather than per tenant?

Every query on one table stalls for forty seconds mid-deploy

Medium · lock contention · ddl · migrations · postgres

During a release on PostgreSQL, every query touching resource times out for about 40 seconds and then recovers with no intervention. The release ran one migration, ALTER TABLE resource ADD COLUMN archived_reason TEXT, and the migration log shows it completing in 6 ms. Unrelated tables showed no change in error rate. Explain how a 6 ms statement caused a 40-second stall, give the ordered checks you would run on a live system to confirm it, and give the migration procedure that prevents a repeat.

Approach

1. Separate the statement's duration from the lock's duration. ADD COLUMN with no default is a catalogue-only change and genuinely runs in milliseconds, but it requires ACCESS EXCLUSIVE, and it cannot acquire that until every transaction already touching the table has finished.
2. Account for the queueing, which is the part that surprises people. A lock request that is waiting blocks later requests for conflicting modes behind it rather than letting them overtake, so one long-open transaction holds the DDL and the DDL holds all the traffic. The stall length is set by the longest open transaction, not by the size of the change.
3. Confirm on a live system in this order: pg_stat_activity for that table ordered by xact_start, looking for the oldest transaction and specifically for state = idle in transaction; then pg_locks where granted = false to find the waiter; then join them on pid to name blocker and blocked. pg_blocking_pids() does that join for you and is the fastest single call.
4. Prevent rather than merely time it better. Set lock_timeout to a second or two on the migration session so the DDL abandons the queue after a bounded wait and is retried, instead of holding it for as long as the oldest transaction lives. Be exact about what that buys: queries arriving during the wait still queue behind the pending ACCESS EXCLUSIVE request, so each attempt costs them up to one lock_timeout of added latency. The outage goes from 40 seconds to about one second per attempt, not to zero. Also run migrations away from deploy-time peaks, and put a statement timeout and an idle-in-transaction timeout on the analytics role that opens the long transactions.
5. Know the lock each change takes, since the mitigation differs by change. A column with a non-volatile default is a metadata-only change from PostgreSQL 11 and still needs the brief ACCESS EXCLUSIVE; an index needs CREATE INDEX CONCURRENTLY, which cannot run inside a transaction block and leaves an INVALID index to drop if it fails; a check or foreign key is added NOT VALID and then VALIDATE CONSTRAINT as a separate statement under a weaker lock.

Follow-up

• The same release also wants NOT NULL on that column. What is the sequence that gets there without a long lock?
• Your lock_timeout retry fails ten times in a row because the analytics transaction is always open. What do you change?
• How does this differ on MySQL with InnoDB online DDL, and what is the equivalent of the waiting-lock queue there?

Suggested practice checklist

For someone fluent in a dynamic language who has shipped real work but has never had to say what the runtime is doing underneath. The week is built on measuring and deliberately breaking things, because the questions that expose this background are the ones where the interviewer asks why a second time.

Day 1 — Measure before reasoning

• Take a slow piece of your own code, write down in advance where you believe the time goes, then profile it and record how wrong the guess was. The cost is usually an allocation you did not notice or an accidental quadratic membership test.
• Replace one list membership test inside a loop with a set and measure at a thousand, ten thousand and a hundred thousand elements, confirming the shape of the curve rather than only that it got faster.
• Write down the three quantities you can now measure instead of assert: wall time, peak memory, and call count for the function you suspected.

Deliverable: A before-and-after profile of real code plus a written note on the size of the gap between the guess and the measurement.

Day 2 — References, copies, and the bugs they produce

• Write the function with a mutable default argument, call it three times, and explain the accumulating result: the default is evaluated once when the function is defined, so every call shares one object.
• Build a nested structure, take a shallow copy, mutate an inner element, and show that both views changed, because a shallow copy duplicates the container and not the elements. Then fix it with a deep copy and state the cost you just accepted.
• Write two functions, one mutating its argument in place and one rebinding the local name, and predict the caller's view of each before running it. That single distinction produces most of the bugs that pass their tests.

Deliverable: Three small programs whose output you predicted correctly before running, each with a one-line statement of the rule underneath.

Day 3 — Types, once, in a language that checks them

• Port one module you have already written, roughly a hundred lines, into a statically typed language, and record every place the compiler demanded an answer your original had left implicit: a value that can be absent, a numeric width, a case never handled.
• Write the same signature in both languages and state what the static one guarantees before the program runs and what it does not, since it will not save you from a wrong algorithm or an index out of range.
• Write the difference between an interface satisfied by declaration and one satisfied structurally, with one case each where the other approach would miss the mistake.

Deliverable: One module in two languages plus a list of the questions the type checker forced you to answer.

Day 4 — Concurrency, starting with what actually runs at the same time

• Run the same CPU-bound function across four threads and four processes and measure both. Under the default CPython build the threaded version will not speed up, because only one thread executes bytecode at a time; the process version will. Check which build you are on first, since free-threaded builds remove that lock and change the result.
• Then run a blocking I/O workload across four threads and measure it speeding up, because the interpreter releases that lock around blocking calls, which is why treating threads as useless is wrong as a general claim.
• Build the lost update: two threads each incrementing a shared counter a hundred thousand times, and show a final value below the expected sum, because an increment is a load, an add and a store and the thread can be suspended between them. Fix it with a lock and then measure what the lock costs.

Deliverable: Three measurements, threads against processes on CPU work, threads on I/O work, and a demonstrated lost update, each with the mechanism written underneath.

Day 5 — Debugging as a procedure rather than an instinct

• Work one real failure as a bisection: find a revision or an input size where it is good and one where it is bad, halve repeatedly, and state the two assumptions bisection needs, that the property changes exactly once across the range and that the test is reliable.
• Minimise one failing input to the smallest version that still fails, and record how many rounds it took.
• Keep a hypothesis log for one bug in three columns, what I believe, what would disprove it, what I observed, and stop yourself the first time you are about to change two things at once.

Deliverable: One bug worked to root cause with a written hypothesis log and a minimised reproducing input.

Day 6 — Tests that catch the bug you are about to write

• Implement an LRU cache with a capacity bound, then write the three test cases that would catch an off-by-one in eviction: insert exactly capacity items and assert nothing was evicted, insert one more and assert the least recently used key is the one gone, and read an old key just before that insert so the eviction victim changes.
• Add a property test comparing your implementation against a deliberately slow reference, an ordered list scanned linearly, over a few thousand random operation sequences, because a slow reference finds the cases you would not have thought to write.
• Write one numeric test that fails under exact equality and passes with a tolerance, and state why the tolerance has to be relative rather than absolute once the magnitudes grow.

Deliverable: An LRU implementation with three boundary tests, one property test against a slow reference, and one tolerance-based numeric test.

Day 7 — Debug something broken, out loud

• Have someone plant three defects in a two-hundred-line program, an off-by-one, a shared mutable state bug, and a wrong error-handling path, then find them while narrating, under a fixed rule: state the hypothesis before touching anything.
• Time each one and record which tool found it, reading, a printed value, a debugger, or a test, because the question asked in interviews is how you would find it rather than what it was.
• Write the sentence you will use when you do not yet know the cause, one that names the next measurement instead of offering a guess.

Deliverable: A recorded debugging session with time-to-find per defect and the method that found each.

The estimate and the day you knew

A slipped date is only a bad story if you sat on it. What matters is what you believed when you gave the number, the signal that told you it was wrong, how many days passed before you said so, and what you cut rather than asking for more time. Scope you defended counts as much as scope you dropped.

Why are you looking to change jobs?

Medium · behavioural and engineering judgement

Approach

1. Give the blast radius: what could have broken, and what you measured.
2. Close with what you would do differently, concretely.
3. Name the disagreement and how you resolved it with evidence.

Follow-up

• What did you decide not to do, and why?
• What would you do differently if you ran that again?

What is your definition of "fairness"?

Medium · behavioural and engineering judgement

Approach

1. Pick a story where you made the decision, not one where you watched it.
2. Close with what you would do differently, concretely.
3. Give the blast radius: what could have broken, and what you measured.

Follow-up

• What did you decide not to do, and why?
• How did you know your change caused the improvement?

How did you manage your time while in college?

Medium · behavioural and engineering judgement

Approach

1. Name the disagreement and how you resolved it with evidence.
2. Give the blast radius: what could have broken, and what you measured.
3. Close with what you would do differently, concretely.

Follow-up

• What would you do differently if you ran that again?
• How did you know your change caused the improvement?