City and County of Honolulu Software Engineer Interview Guide

The 60-second brief

The City and County of Honolulu is a municipal government, and candidates describe the Software Engineer role as building, testing and maintaining applications for city departments and the residents who use city services. Reported work areas include data management systems, applications that non-specialists can use, and protecting sensitive information. The role also involves gathering requirements from different departments, updating applications that already run, and taking part in code reviews.

This guide is for engineers applying to the City and County of Honolulu as a Software Engineer, whether you come from a private-sector team or from government work. It covers the process candidates describe (online application, a long wait, an in-person panel, background checks), the five technical questions reported for this employer, and the stack and evaluation areas listed for the role: Java, C# or Python, web technologies, SQL and NoSQL, REST APIs, Agile and Waterfall, and application security.

Candidates describe an online application with resume, diploma and transcripts, a wait of about two months, then an in-person panel interview that mixes technical and behavioral questions, followed by background checks. No separate round names or order are reported beyond that.

Focus areas

• Software Engineering Fundamentals
• Technical Interview Preparation
• Problem Solving

Overview

The reported process has few stages and a long wait. You apply online with your resume, diploma and transcripts, wait roughly two months, and may then be invited to an in-person interview with a panel. Background checks come after. Because the gap between application and interview is long, you can prepare in a planned way instead of cramming: the plan below is built to be started when the invitation arrives, and it is also fine to work through it earlier and refresh it.

The technical questions candidates report are broad rather than puzzle-shaped: which languages you are strongest in and why, how object-oriented and functional programming differ, a challenging technical problem you solved, how you debug and test, and which tools you use for version control and collaboration. None of them has a single correct answer, so the preparation is about having a precise, experience-backed answer ready for each, with concrete examples from code you have written, rather than memorising definitions.

The role description adds the stack and the breadth to cover. Candidates describe Java, C# or Python as the expected languages, HTML, CSS and JavaScript frameworks for web work, SQL and NoSQL databases, REST API development and third-party integration, Agile and Scrum alongside Waterfall, and application security. Cloud platforms (AWS, Azure), DevOps practice and public-sector software experience are listed as nice-to-have. Be ready to connect each of these to something you have shipped, and to say plainly where your experience stops.

Reported evaluation areas also include problem-solving, communication and culture fit, and the day-to-day work involves gathering requirements from several departments. Prepare at least one story about turning a vague request from a non-technical colleague into working software, and one about a code review. Candidates also advise practising coding and system design aloud, so keep a small amount of whiteboard-style practice in your week even though no coding stage is formally reported.

Where to focus your preparation

Only one stage is reported for this employer: an in-person panel interview with technical and behavioral parts. The entry below is a preparation focus for that panel, not a sequence of named rounds.

Preparation focus

Candidates describe an in-person interview run by a panel, reached after the online application and a long wait, with both technical questions and behavioral assessments, and background checks afterwards. No round names, order or per-stage format are reported, so treat this as one preparation block. Work the five reported technical questions aloud, prepare a small number of strong stories, and confirm with the recruiter how many people will be on the panel and whether you should expect to write code on the day.

What to demonstrate

• Command of the languages the role lists (Java, C# or Python) and the ability to explain why you choose one
• Software fundamentals: object-oriented versus functional design, debugging, testing and version control
• Working knowledge of the listed evaluation areas: SDLC methods, SQL and NoSQL, REST integration and application security
• Communication with a panel and teamwork across departments, which candidates report alongside the technical content

How to prepare

• Write a short, honest answer on your strongest language: three concrete strengths, one real limitation and a case where you would pick another of Java, C# or Python
• Implement one small task twice, in an object-oriented style and in a functional style, and list what changed about state, mutation and testing
• Write out your debugging routine as steps (reproduce, isolate, form a hypothesis, fix, add a regression test) using a real bug you fixed
• Prepare one challenging-problem story and one cross-department requirements story, each with the outcome, and rehearse them out loud

Where candidates lose points

Answering the favourite-language question with a list of languages and no depth

Choose one or two languages from Java, C# or Python (or whatever you use best) and prepare specifics: a feature you rely on, a project where it was the right tool, and one limitation you have worked around. A reason tied to something you built is stronger than a ranking.

Presenting object-oriented and functional programming as opposing camps with textbook one-liners

Contrast them on concrete properties: mutable state versus immutable data, inheritance versus composition and function composition, and how each affects testing. Point out that Java, C# and Python all support both styles, and show a small example from your own code where you mixed them.

Telling the challenging-problem story as a list of tools used, with no cause, no decision and no result

Structure it as situation, the specific cause you found, the options you weighed, what you did, how you verified the fix and what you changed afterwards so it did not recur. Pick a problem where you made a decision, not one the team solved around you.

Saying you debug by using the debugger and test by running the program

Describe a sequence: reproduce the failure, narrow the scope, read logs or state, test a hypothesis, fix, then add a regression test. Name the test layers you use (unit, integration, end to end) and what each catches, and say how you review other people's changes.

Treating the panel as purely technical and having no story about working with non-engineers

Candidates report behavioral and culture-fit questions next to the technical ones, and the role involves gathering requirements from departments. Prepare an example of clarifying an unclear request, one about a code review you gave or received, and one where you handled sensitive data carefully.

Questions & practice

Collapse a redelivered event batch into per-aggregate high-water marks

Easy · hashing · at-least-once · aggregation

You drain a batch of up to 5,000,000 events, each (aggregate_id BIGINT, aggregate_version INT, event_type, payload). The log guarantees order within one aggregate only; the batch merges 64 partitions, and a relay failover has redelivered a range, so an older version for an aggregate can appear after a newer one. Given a map of last_applied_version per aggregate, produce the events worth applying, at most one per (aggregate_id, version), plus the count discarded. Target O(n) time. State the memory for 2,000,000 distinct aggregates and what you do when it does not fit.

Approach

1. One pass, one hash map from aggregate_id to the highest version kept, and a discard counter. An event whose version is at or below last_applied_version for its aggregate is dropped without further work, which is the whole reason the event carries its version rather than a delta. O(n) expected time, O(d) space in distinct aggregates.
2. Keep the maximum, never the last occurrence. The redelivered range means the final appearance of an aggregate in the batch can be an older version than one seen earlier in the same batch, so last-wins applies stale state over newer state and the projection regresses with no error anywhere.
3. Cost the memory instead of calling it large: an 8-byte key plus a 4-byte version is 12 bytes of payload, and an open-addressed table held at a 0.7 load factor costs roughly 17 bytes per entry before per-slot metadata, so 2,000,000 aggregates is tens of megabytes in a native layout and several times that in a runtime that boxes both key and value.
4. If the distinct set exceeds memory, partition on hash(aggregate_id) mod P and reduce each partition independently. Every event for one aggregate hashes to the same partition, so the per-partition result is exact and the merge is concatenation rather than a second reduction.

Follow-up

• The payload is a patch rather than a snapshot, so applying only the highest version loses the intermediate changes. What changes in your reduction?
• How do you detect that version 7 arrived while version 6 was never delivered, and what should the consumer do about the gap?

Find overlapping job attempts and peak concurrency from lease records

Medium · sweep line · intervals · leases

A day of job_run history yields about 50,000,000 attempt records: (job_run_id, job_type, attempt, started_at, finished_at which is NULL when the worker died, lease_expires_at). Leases expire on a clock, so a job that outran its lease ran twice. Produce (a) every job_run_id whose attempts overlapped in wall-clock time and (b) the peak number of simultaneously running attempts per job_type with the minute it occurred. Target O(n log n). State how you treat a NULL finished_at and what clock skew does to your answer.

Approach

1. Define the interval before sorting anything: an attempt occupies [started_at, COALESCE(finished_at, lease_expires_at)). finished_at is observed and lease_expires_at is only a promise, so every attempt without a finish contributes an estimate and the whole result is a lower bound on overlap rather than an exact count.
2. For peak concurrency, sweep: emit 2n endpoints, sort by (timestamp, kind) with ends ordered before starts at equal timestamps, then walk the sequence maintaining a counter per job_type and record each type's maximum with its timestamp. O(n log n) dominated by the sort, O(n) space, or O(1) extra if the sort is external and the walk streams.
3. For overlap detection, do not compare attempts pairwise. A single global sort by (job_run_id, started_at) gives both the grouping and the order; within a group, keep the maximum end seen so far and report an overlap exactly when the next start is less than that running maximum, which is one linear pass after the sort.
4. Half-open intervals matter and are easy to get wrong: with closed intervals an attempt ending at the same millisecond another begins reads as concurrency two, and across 50,000,000 records that artefact swamps the real signal.

Follow-up

• A handler is not idempotent and you have found 400 overlapping jobs. Which of them actually caused damage, and what would you query to find out?
• Peak concurrency for one job_type is 4 against a configured cap of 4. Is the cap working, or is the data hiding attempts that never started?

Archive a resource graph without breaking live references or recursing

Medium · graph traversal · topological order · tenant isolation

Resources reference other resources within a tenant; for the largest tenant the reference table holds up to 2,000,000 nodes and 8,000,000 edges. Archiving a resource must archive everything reachable from it that nothing outside the set still references, refuse when a live external referrer exists, and terminate when references form cycles, which they legitimately do. Produce the archive order and the refusal list, targeting O(V+E). Say what stops the traversal crossing a tenant boundary, and why recursion is the wrong control structure at this size.

Approach

1. Load the subgraph with the tenant predicate on both endpoints of the edge, not only on the side you started from. Scoping the left table alone is the classic cross-tenant leak: one mis-entered edge then pulls another tenant's resources into the traversal and, worse, into the archive.
2. Traverse iteratively with an explicit stack. A 2,000,000-node graph can hold a chain deep enough to exhaust a native stack in the low tens of thousands of frames, and that failure is a process crash rather than an error you can return.
3. Treat cycles as data rather than corruption: compute strongly connected components with Tarjan in O(V+E) using its own explicit stack, then condense. The condensation is a DAG, so a topological order over it gives the archive order, and every member of a component archives in one transaction because no order within a cycle is valid.
4. Decide refusals with reverse edges. A candidate is archivable only if every in-edge originates inside the candidate set, so build the transpose or count in-degrees restricted to the visited set, and emit each blocked resource with the id of the external referrer, which is the only part of the answer an operator can act on.

Follow-up

• The graph is read in one query and the archive writes a minute later. What can change in between, and how do you make the write safe?
• The candidate set is 400,000 resources. Is that one transaction, and if not, what does a half-finished archive look like to a reader?

Hold a per-tenant active cap against concurrent creates

Hard · write skew · isolation levels · constraints

A tenant on the standard plan may hold at most 50 resources with status='active'. The create handler runs SELECT count(*) FROM resource WHERE tenant_id = $1 AND status = 'active', compares to 50, then inserts. Two creates arrive 3 ms apart on different instances and the tenant lands at 51. Name the anomaly, say whether PostgreSQL 16 READ COMMITTED or REPEATABLE READ prevents it and why, then give an implementation that holds the cap at READ COMMITTED with the exact statements. Finally, say what changes when the cap is 'at most one running export per tenant' on job_run.

Approach

1. Name it: write skew. The two transactions read an overlapping set and write disjoint rows, so there is no row-level conflict for the engine to detect and each commit is individually legal.
2. Rule out the levels precisely. READ COMMITTED takes a fresh snapshot per statement and takes no lock on the counted rows, so both see 49. PostgreSQL's REPEATABLE READ is snapshot isolation: it removes non-repeatable reads and phantoms within the snapshot but still admits write skew, because the anomaly is not a re-read of a changed row, it is a read of a set that a concurrent transaction invalidates. Only SERIALIZABLE closes it, by tracking the read dependency and aborting one transaction with SQLSTATE 40001 — a guarantee that exists only if the application re-runs the whole transaction from the read.
3. Convert the set predicate into a single-row conflict: keep tenant.active_resource_count and run UPDATE tenant SET active_resource_count = active_resource_count + 1 WHERE tenant_id = $1 AND active_resource_count < 50 in the same transaction as the INSERT. Zero affected rows is the cap, returned as 409. The row lock serialises the decision at any isolation level, and contention is bounded to one tenant's row — which is also the fair-scheduling unit, unlike a global counter that would convoy every tenant behind one row.
4. State the cost you just took on: a counter is a second source of truth that can drift, so every path that changes status must adjust it inside the same transaction, and a periodic reconciliation has to exist, with resource_revision as the authority for what the count should have been.

Follow-up

• A resource moves from archived back to active. Which statements change, and what breaks if the counter update and the status change land in different transactions?
• The cap becomes plan-dependent and a plan can change mid-month. Where does the number 50 live, and who reads it?

Stop tag and share joins from fanning out a page

Medium · join fan-out · semi-joins · many-to-many

resource_tag is (resource_id, tag_id) with PK (resource_id, tag_id); resource_share is (resource_id, shared_with_user_id, permission). The tagged-and-shared listing inner-joins resource to both, filters tenant_id, tag_id = ANY($2) and shared_with_user_id = $3, orders by updated_at DESC and takes 50. Pages come back with fewer than 50 distinct resources and the total in the header is far too high. Explain the row multiplication, rewrite both the page query and the count query so each is correct, and name the index each one needs. PostgreSQL 16.

Approach

1. Do the arithmetic against the predicates that are actually there. An inner join emits one row per matching child row, and both joins are filtered: tag_id = ANY($2) admits only the requested tags, shared_with_user_id = $3 admits one user's share rows. So a resource holding three of the requested tags and shared with $3 once yields three rows, not one — the multiplier is its count of matching tags times its share rows for that single user, and that second factor is 1 unless the table admits duplicate (resource_id, shared_with_user_id) pairs. LIMIT 50 then limits rows rather than resources, and COUNT(*) counts pairs — the header is the product, not the population.
2. Reject DISTINCT as the fix. It deduplicates after the product has been built, so the planner must materialise and sort the fanned-out set before the LIMIT can apply, and it leaves any SUM or AVG in the same select list wrong.
3. Rewrite both filters as semi-joins, keeping resource as the only row source: AND EXISTS (SELECT 1 FROM resource_tag rt WHERE rt.resource_id = r.resource_id AND rt.tag_id = ANY($2)) and the same shape against resource_share. A semi-join stops at the first match per resource and preserves the driving index order, so ORDER BY updated_at DESC, resource_id DESC LIMIT 50 still stops after 50 rows.
4. Count with the same predicates and no join at all: SELECT count(*) FROM resource r WHERE r.tenant_id = $1 AND r.status = 'active' AND EXISTS (...) AND EXISTS (...). Nothing multiplies a resource, so the number is the population.

Follow-up

• The filter changes from 'any of these tags' to 'all of these tags'. Rewrite it and state what it costs relative to the ANY form.
• A resource can be shared with the same user twice under different permissions. Does your count change, and should it?

What programming languages are you most proficient in, and why?

Medium · Technical / Domain Questions

Approach

1. Lead with one or two languages, ideally among the Java, C# or Python that candidates report the role listing, and back each with evidence: what you built in it, how large the codebase was, and how long you maintained it.
2. Explain the 'why' in technical terms. Java and C# are statically typed, so the compiler catches type errors before runtime and large refactors are safer. Python is dynamically typed, which makes iteration fast and gives you a large library ecosystem, with type hints plus a checker such as mypy for safety.
3. Show depth beyond syntax with one pitfall you understand: Java keys in a HashMap need equals and hashCode overridden together; C# LINQ queries are deferred and run again on every enumeration; Python default arguments are evaluated once, so a mutable default is shared across calls.
4. Know the runtime trade-offs. In the default CPython build the GIL stops threads from running Python bytecode in parallel, so CPU-bound work goes to multiprocessing, while the JVM and .NET run threads in parallel. Then name your supporting skills (SQL, JavaScript for web front ends) and be honest about where your level drops.

Follow-up

• Which feature of that language would you avoid in a shared codebase, and what bug has it caused you?
• If you had to build a small REST service backed by a SQL database, which language and framework would you pick, and why?
• How would you get productive in C# if most of your experience is in Java, or the other way round?

Can you explain the differences between object-oriented and functional programming?

Medium · Technical / Domain Questions

Approach

1. Define both. Object-oriented programming bundles state with the methods that change it: encapsulation, inheritance, and polymorphism through interfaces or virtual dispatch. Functional programming builds programs from functions: pure functions, immutable data, first-class and higher-order functions, and composition.
2. Put the main difference in where state lives. OOP changes encapsulated state through methods. FP avoids shared mutable state and returns new values, so a pure function gives the same output for the same input, is easy to unit test, and is safe to run concurrently.
3. Name the extension trade-off (the expression problem). In OOP, adding a new type is easy because you write one new class, but adding a new operation touches every class. With functional data types and pattern matching, adding an operation is one new function, but adding a new variant touches every function.
4. Show that mainstream languages mix the two. Java has had lambdas and streams since Java 8; C# has LINQ, records and pattern matching; Python has comprehensions, map/filter and functools. A practical pattern is pure transformation logic inside class-based services, with database and network I/O kept at the edges.
5. State the costs. Immutability means copying unless the data structure shares structure. Deep inheritance trees produce the fragile base class problem, so prefer composition and small interfaces.

Follow-up

• What makes a function pure? Is a function that writes a log line pure?
• Rewrite a loop that totals the amounts of paid invoices in a functional style, in Java streams or Python.
• When has inheritance hurt you, and how would composition have changed that design?

Describe a challenging technical problem you faced and how you resolved it.

Medium · Technical / Domain Questions

Approach

1. Pick a problem whose technical root cause you found yourself, not a team-wide saga. Open with the symptom and its impact in concrete terms: error rate, slow page, failed job, number of users affected.
2. Walk through the investigation in order: the hypotheses you formed, the evidence that ruled each one in or out (logs, metrics, a reliable reproduction, a profiler, a database query plan), and the root cause you landed on.
3. Be specific enough to survive a deeper question. For example: an N+1 query pattern I spotted in the SQL logs and replaced with a join or a batched fetch, or duplicate rows from a race condition that I fixed with a unique constraint and an idempotent upsert.
4. Explain why you chose that fix over the alternatives you considered, how you verified it after release, and what stops it coming back (a regression test, an alert, a runbook entry).
5. Say 'I' for your own actions and credit others accurately. A story that never names a root cause, or that blames a colleague or vendor, sinks the answer.

Follow-up

• What did you try first that did not work, and how did you know it had not worked?
• How did you confirm the fix in production rather than only on your machine?
• What would you do differently if you hit the same problem today?

What tools do you use for version control and collaboration?

Medium · Technical / Domain Questions

Approach

1. Name Git and the hosting platforms and trackers you have actually used (GitHub, GitLab, Bitbucket or Azure DevOps; Jira or Azure Boards for Agile/Scrum work), then describe your workflow: short-lived feature branches off main, small commits with clear messages, a pull request per change.
2. Show command-level understanding. A merge keeps the history and adds a merge commit. A rebase rewrites your commits onto a new base for a linear history, so never rebase a branch others have already pulled. git revert adds a new commit that undoes a change and is safe on shared branches; git reset moves the branch pointer and rewrites history.
3. Connect the tools to code review, which is among the responsibilities candidates report for this role. Use branch protection, required reviewers, and CI checks (tests, lint, build) that must pass before merge. Link commits and pull requests to tracker tickets so every change traces back to a requirement.
4. Mention security hygiene. Keep credentials out of the repository with .gitignore and a secrets store. If a secret is pushed, rotate it at once: deleting it from history does not help, because clones and forks may already hold it.
5. If asked, compare branching models. Trunk-based development with feature flags keeps integration continuous. Git Flow's long-lived develop and release branches suit scheduled releases but cause more merge conflicts.

Follow-up

• A teammate pushed a database password to the shared repository. What do you do, step by step?
• Do you merge or rebase, and when would you choose each?
• What do you look for when reviewing a large pull request, and how do you give feedback on it?

How do you approach debugging and testing your code?

Medium · Technical / Domain Questions

Approach

1. Reproduce before fixing. Shrink the failure to the smallest input or steps that trigger it, and capture that as a failing automated test, so the fix is proven and the bug cannot quietly come back.
2. Narrow it down systematically. Read the full stack trace and logs, form one hypothesis at a time, and test it with a debugger breakpoint or targeted logging. When a regression appeared between two known versions, git bisect binary-searches the commits, so n commits need about log2(n) checks, and git bisect run can automate them with a test script.
3. Fix the root cause, not the symptom, and search the codebase for the same pattern elsewhere. Then explain the testing layers: many fast unit tests (JUnit, xUnit/NUnit or pytest), fewer integration tests against a real database and API, and a small number of end-to-end UI tests.
4. Cover edge cases on purpose: empty and null input, boundary values, invalid or malicious input, missing permissions and failed authentication (which ties to the security requirements candidates report). Mock only at external boundaries, and run the whole suite in CI on every pull request. Treat coverage as a signal, not a target.

Follow-up

• A bug appears only in production and never locally. What do you check, and in what order?
• How do you test code that calls a third-party API you do not control?
• What makes a test flaky, and how have you fixed one?

Suggested practice checklist

Seven days built around the single reported panel: the five reported technical questions get their own days, the listed stack (SQL and NoSQL, REST, security, Agile and Waterfall) gets a breadth day, and the last day is a full spoken mock.

Day 1 — Inventory your experience and confirm the format

• List every project where you used Java, C# or Python, plus any HTML, CSS or JavaScript framework, SQL or NoSQL database, and REST API work
• Rate yourself on the listed evaluation areas: Agile and Waterfall, database management, APIs and integration, and security
• Write the questions to ask the recruiter: panel size, whether you will write code, and what to bring

Deliverable: A one-page experience inventory, a rated checklist of the four evaluation areas, and a list of format questions for the recruiter.

Day 2 — Languages: choose and defend

• Write your answer to "What programming languages are you most proficient in, and why?" with three strengths, one limitation and a project for each language you name
• Prepare a comparison of Java, C# and Python (or the ones you know) for a typical city application, so you can say when you would switch
• Say the answer aloud and tighten any sentence that is a claim without an example

Deliverable: A written and spoken answer on language proficiency that names concrete projects and a limitation.

Day 3 — Object-oriented versus functional programming

• Pick a small task, such as filtering and totalling a list of records, and write it in an object-oriented style and a functional style in your strongest language
• Write a contrast table covering state, mutation, inheritance versus composition, testability and concurrency
• Prepare one example of where you used each style in real work and why

Deliverable: Two short implementations of the same task and a contrast table you can explain from memory.

Day 4 — Debugging, testing and collaboration tooling

• Write a step-by-step debugging walkthrough from a real bug, from first report to regression test
• List the test layers you use, which tools you run them with, and what each one catches
• Describe your version-control workflow (branching, pull requests, reviews) and the collaboration tools you have used, then answer both questions aloud

Deliverable: A debugging walkthrough, a test-layer list and a written version-control and collaboration answer.

Day 5 — The challenging-problem story and teamwork stories

• Write the challenging technical problem story in situation, cause, action, verification and follow-up form
• Write a second story about gathering requirements from a non-technical stakeholder

Deliverable: Two written stories with outcomes, each rehearsed aloud, plus notes from one behavioral practice question.

Day 6 — Stack breadth: SQL, REST, security and methods

• Work the two SQL practice questions on concurrent creates and join fan-out to refresh how queries behave under load and with many joins
• Write short answers on REST design and third-party integration: authentication, error handling, retries and versioning
• Write short answers on application security (input validation, parameterised queries, access control, protecting personal data) and on Agile versus Waterfall

Deliverable: Two worked SQL practice questions and short written answers on REST integration, security and delivery methods.

Day 7 — Full spoken mock of the panel

• Answer all five reported technical questions aloud in sequence, then your two stories, with someone playing the panel if you can
• Write two questions to ask the panel about the team and the applications it maintains

Deliverable: A completed spoken mock, a list of weak answers to tighten and two questions for the panel.

Stories for a public-sector engineering panel

Candidates report behavioral assessments and a culture-fit element alongside the technical questions, and the role description centres on collaboration with different departments, code reviews and keeping existing applications updated. Prepare stories that show you turning unclear requests into working software, handling feedback, and treating security and data with care.

1. Describe a challenging technical problem you faced and how you resolved it.
2. How do you approach debugging and testing your code?
3. Tell me about a time you gathered requirements from people who were not engineers and changed what you built because of it.
4. Describe a code review where you gave or received feedback that changed the code. What changed and why?
5. Tell me about a time you worked with sensitive data or a security concern and what you did about it.
6. Why do you want to build software for public services, and what in your background supports that?

Reverse your own decision and price the reversal

Medium · reversibility · measurement · migrations

Describe a technical decision you made and later reversed. Pick one that cost something: a service you split and merged back, a cache you added and removed, an index you created that pushed the planner onto a worse plan, a projection you rebuilt from scratch. State what you believed when you decided, the measurement that changed your mind, how long the wrong version ran in production, and what the reversal cost in migrations, dual writes, and a deprecation window for callers you did not own.

Approach

1. State the original rationale without irony, in the version you would still defend given what was known then. If it is not defensible, the story is about carelessness rather than judgement, and a different example serves you better.
2. Give the measurement that moved with a before and after: the p99 that did not improve, the cache hit rate that sat at 40%, the plan that flipped to a sequential scan once the table passed a size you can name.
3. Cost the reversal in steps, not adjectives: expand-and-contract deploys, the dual-write window, the callers who had to be notified, the rows already written in the wrong shape that had to be backfilled or abandoned.
4. Distinguish reversal from rewrite by naming what you kept. Most good reversals preserve the schema or the interface and undo one decision inside it, which is also why they were affordable.

Follow-up

• What in that decision was irreversible, and did you know it was irreversible when you made it?
• How did you tell the people who had already built on top of the original decision?

Unblock an engineer without taking the keyboard

Easy · mentoring · leases · at-least-once

A teammate has spent two days on a job handler that occasionally writes duplicate rows. They are certain the queue is delivering twice by mistake. You suspect a lease expiring under a slow handler, so the job is running concurrently with itself. Describe how you have unblocked someone in this position: what you asked before offering a hypothesis, what you showed them rather than told them, and what you left them owning. Then say what you would do if their theory turned out to be the right one.

Approach

1. Ask before diagnosing, and ask for things answerable from data they already have: the attempt count on the job rows that produced duplicates, the handler's observed duration against its lease expiry, and whether the duplicate rows share a natural key that a unique constraint could have caught.
2. Teach the shape rather than the answer. A lease cannot distinguish a dead worker from a slow one, so a handler that outruns its lease is running twice by design, and deploys deliver the other half by killing handlers mid-run on every rollout. Both of their candidate theories produce identical duplicate rows, which is why the evidence has to come from timings rather than from argument.
3. Hand over a checklist they execute: a natural key on every write the handler performs so the second copy collides rather than appends, the record of intent written before any external effect, a lease heartbeat while running, and the metric that shows it working.
4. Keep ownership with them deliberately. Pair on the first write, then step back; if you finish it yourself you have closed one ticket and left the same person stuck on the next redelivery.

Follow-up

• How would you distinguish a genuine double-delivery from a lease expiry using only the data already stored?
• Their handler calls an external endpoint before recording that it did. What do you tell them to change first?

Estimate work you have never done and defend the range

Hard · estimation · backfills · expand-contract

You are asked to estimate a change you have never attempted: add a column to a 100-million-row table, populate it, move reads across, and drop the old shape. Give a range with the assumptions that generate it, including batch size, the signal your backfill throttles on, and wall-clock hours, and name the three unknowns that would move the number most. Then describe a real estimate you gave under comparable ignorance: how you expressed its uncertainty, what you committed to, and how wrong you turned out to be.

Approach

1. Decompose into independently deployable steps before estimating anything: add the column nullable, write both shapes, backfill in batches, verify, move reads, stop writing the old shape, drop it. That is four deploys spread over days, and the calendar estimate is dominated by them rather than by the loop's runtime.
2. Do the arithmetic aloud for the part that has arithmetic in it: batch size times number of batches times per-batch duration, at a write rate the primary can absorb alongside roughly 1.2k writes per second of production traffic. The loop is throttled by replication lag and lock waits, not by how fast it can issue statements.
3. Price the schema step by its lock rather than its statement duration. In PostgreSQL an ALTER TABLE taking ACCESS EXCLUSIVE waits for every open transaction on that table while later queries queue behind it, so a millisecond change issued during a thirty-second analytics query stalls that table for thirty seconds. Adding a nullable column with a non-volatile default avoids a rewrite from version 11; a new index wants CREATE INDEX CONCURRENTLY, which cannot run inside a transaction block and leaves an invalid index behind if it fails.
4. Express the answer as a range whose endpoints each trace to a stated assumption, then name the cheapest experiment that collapses it, which is almost always running one real batch against the real table and multiplying.

Follow-up

• How do you verify the backfill genuinely finished, given rows written by production traffic while it ran?
• Where does the backfill resume from after a worker is killed mid-batch, and what makes that resume point trustworthy?