Interview guides · Adapt Forward

Adapt Forward Software Engineer Interview Guide

Adapt Forward Software Engineer interview guide: the 4 reported rounds, coding and system design questions, worked approaches and a 7-day prep plan.

The 60-second brief

The role description places Software Engineers in Baton, Adapt Forward's product development group, and ties the work to Ryder's customers in transportation and logistics. It describes products ranging from user-facing applications to the data platforms behind analytics, all in the setting of supply-chain automation. It also lists collaboration with product managers, designers and data scientists, plus code review. For interview purposes, examples built on shipments, inventory, tracking events and carrier data sit closer to this work than abstract ones.

This guide is for engineers interviewing for the Software Engineer role at Adapt Forward. The requirements candidates describe are Python, Java or Go, software design and architecture experience, and team collaboration. Cloud (AWS, Azure), microservices and APIs, Agile tooling such as JIRA and Confluence, and a logistics or supply-chain background are listed as pluses. Candidates describe the typical hire as having 5+ years of experience, including mentoring. The guide covers all four reported rounds and gives specific guidance for each of the 25 reported questions: five coding, five system design, five engineering-practice, five problem-solving cases and five behavioral.

Candidates report four rounds over roughly 3-5 weeks: an Initial Screening Call, then Technical Interviews that may include coding challenges or system design, then Team Member Meetings and Leadership Meetings.

Focus areas

Platform / Horizontal Services Product Strategy
Product Roadmap Ownership (end-to-end)
0→1→Scale Product Lifecycle

Overview

The reported questions fall into five groups of five. The coding questions are classic fundamentals: reversing a linked list, binary search, sorting faster than O(n^2), dynamic programming and recursion. The system design questions lean toward logistics: a real-time inventory system and a REST API for a logistics application, plus data consistency, high availability and SQL vs NoSQL. The engineering-practice questions cover design patterns, microservices, a production bug, testing and version control. The problem-solving cases are performance tuning, data anomalies, user feedback, troubleshooting and feature prioritization, and the last group is behavioral questions about leadership and teamwork.

Because the coding questions are well-known problems, getting the answer right is only the starting point. What sets a strong answer apart is clean code written in one language, with correct edge cases and a stated time and space complexity. For a linked list that means the empty and single-node cases. For binary search it means the loop invariant and the overflow-safe midpoint. Pick Python, Java or Go (the languages candidates report for the role) and use the same one in every practice session.

The design questions reward domain grounding. An inventory system that cannot stop two orders from claiming the last unit is incomplete. So is a logistics API with no idempotency on shipment creation. Prepare concrete components: an atomic conditional update or ledger for stock, an outbox for reliable events, cursor pagination and webhooks for tracking. Then practise explaining the trade-off behind each one, rather than reciting a generic template.

Two of the four reported rounds are conversations with team members and with leadership. Candidates describe them as being about collaboration, fit and culture alignment, and the role description mentions mentoring junior engineers. So spend real preparation time on five or six behavioral stories, not just on algorithms. Each story should show a decision you made yourself, the trade-off behind it and a measurable result.

The Adapt Forward interview process

Below are the four stages in the order candidates report them. Only the Technical Interviews stage is reported to include coding or system design. The other three are described as conversations about background, collaboration and culture.

Initial Screening Call

Candidates describe this as a preliminary call about your background and whether you fit the role. Nothing technical is reported for it. Treat it as the place to connect your experience to the requirements candidates cite: Python, Java or Go, software design and architecture, and team collaboration. Candidates also report that questions can vary by team, so ask which team you are being considered for and what the rest of the loop covers. Then focus the following rounds' preparation on that team.

What to demonstrate

Whether your background matches the reported requirements: Python, Java or Go, and software design experience
Your background and fit for the role, as candidates describe the call

How to prepare

Prepare a short career walkthrough built around two or three systems you built, naming the language, the scale and your own part in each.
Prepare one sentence on why logistics software interests you, tied to something concrete such as tracking, inventory or routing data.
Ask which team you are interviewing for and whether the technical stage leans toward coding, system design or both.
Have your location constraints ready. Candidates report a hybrid arrangement, so know which office days work for you.

Technical Interviews

Candidates report technical interviews that may include coding challenges or system design discussions. This is where the reported coding questions (linked lists, binary search, sorting, dynamic programming, recursion) and design questions (inventory, a logistics REST API, consistency, availability, data stores) get practised. Prepare for both formats. When coding, say the approach and its complexity before you write, write working code in one language, and test it against edge cases out loud. In design, start from requirements and the read/write patterns, then go deep on the hardest part of the problem, such as overselling stock or retried API calls.

What to demonstrate

Working code with correct edge cases and stated time and space complexity
System design reasoning: components, data model, and the trade-offs between consistency, availability and cost
Engineering judgement on testing, code quality and production debugging

How to prepare

Write from memory, in Python, Java or Go: iterative linked-list reversal, lower-bound binary search, merge sort and a memoised dynamic program, each with tests.
Work the real-time inventory design end to end, including the atomic reservation update and the event stream, then the logistics REST API with idempotency keys and pagination.
Prepare a short explanation each for the saga and outbox patterns, multi-AZ failover, and when you would choose a relational store over a NoSQL one.
Rehearse one production-bug story and your testing approach as concrete, numbered steps.

Team Member Meetings

Candidates describe meetings with team members focused on collaboration and fit within the team. No coding is reported for this stage. The role description candidates cite involves working with product managers, UX/UI designers and data scientists, plus code reviews and retrospectives, so prepare examples of how you work on a team day to day: giving and receiving code review, handling disagreements and sharing knowledge. Use specific episodes rather than general statements about valuing teamwork.

What to demonstrate

Collaboration and fit within the team, as candidates describe the stage
Preparation focus: examples of working with peers and cross-functional partners, including code review and disagreements

How to prepare

Prepare a story about a code review disagreement: what the issue was, the evidence you used and how it was resolved.
Prepare an example of working with a product manager, designer or data scientist where a requirement changed because of your input.
Be ready to describe your Git workflow concretely: branch strategy, PR size, how you handle conflicts and reverts.
Prepare two questions for the team about how they run code review and retrospectives.

Leadership Meetings

Candidates describe interviews with leadership that assess alignment with the organisation's culture. Nothing technical is reported. Prepare behavioral stories about ownership, running projects under pressure, stakeholder trade-offs, mentoring and adapting to change, since the reported behavioral questions cover those themes. Each story should show a decision you made, the alternative you rejected and a result you can measure. Also prepare questions about how Baton chooses which products to build and how engineers feed into those decisions.

What to demonstrate

Alignment with the organisation's culture, as candidates describe the stage
Preparation focus: ownership shown through concrete past decisions
Preparation focus: how you handle changing priorities and ambiguity

How to prepare

Write five STAR stories (deadline, stakeholder conflict, mentoring, adapting to change, feedback culture), each with a number in the result.
For each story, note the trade-off you made and what you would change now.
Prepare a story about learning a new technology quickly to deliver a project.
Prepare two questions for leadership about product direction and how engineering input shapes the roadmap.

Where candidates lose points

Reversing the linked list but overwriting curr.next before saving it, or giving no time and space complexity for the coding questions

For every reported coding question (linked list, binary search, sorting, dynamic programming, recursion), say the approach and its complexity before you write anything. Then trace a three-element example and the empty case out loud after you finish. Practise in one language (Python, Java or Go) until the pointer updates and the loop bounds come without thinking.

Designing the real-time inventory system as a CRUD table with no answer for two orders racing for the last unit

Make concurrency the centre of the inventory design: an atomic conditional update or ledger-based reservation, idempotency keys for retried device writes, and a transactional outbox for events. Prepare the same depth for the logistics REST API: idempotent POST, 409 on illegal state changes, and webhooks for tracking.

Answering the microservices or SQL vs NoSQL questions with a list of advantages and no costs

For every architecture question, give the cost alongside the benefit: network failure and sagas for microservices, weak joins and access-pattern lock-in for NoSQL, failover data loss for high availability. Finish with a decision rule tied to a logistics example, such as orders in PostgreSQL and GPS pings in a wide-column store.

Telling a production-bug or performance story with no evidence trail: 'we looked into it and found a config issue'

Rehearse the debugging and troubleshooting answers as a sequence of steps: scope, what changed, the metrics and traces you read, the hypotheses you ruled out, the precise root cause, the mitigation, the fix and the prevention. Use your own first-person actions throughout.

Behavioral answers about running a project, mentoring or stakeholder conflict where 'we' did everything and nothing was decided

For the reported behavioral questions, write each story with one explicit decision you made, the alternative you rejected, and a numeric result. Check that mentoring stories show the mentee's measurable growth and that deadline stories show a scope trade-off rather than overtime.

Questions & practice

Write a function to reverse a linked list. What are the time and space complexities?

Medium · Coding / Algorithms

Approach

Iterative, three pointers: prev = None, curr = head. While curr: nxt = curr.next; curr.next = prev; prev = curr; curr = nxt. Return prev as the new head. O(n) time, O(1) extra space.
Save next before rewiring, or you lose the rest of the list. Walking through a three-node example out loud catches this.
Edge cases: empty list (return None), a single node, two nodes. Test by building a list from an array and converting it back.
Recursive version: the base case comes first, if head is None or head.next is None: return head. Then new_head = reverse(head.next), head.next.next = head, head.next = None, and return new_head (the same node at every level). It is O(n) time but O(n) stack space, and in Python it hits the default recursion limit (about 1000) on long lists, so prefer the iterative version in production.

Follow-up

Reverse only the nodes between positions m and n.
Reverse the list in groups of k.
Check whether a linked list is a palindrome in O(1) extra space.

How would you implement a binary search algorithm in your preferred programming language?

Medium · Coding / Algorithms

Approach

Classic search on a sorted array: lo = 0, hi = n - 1; while lo <= hi: mid = lo + (hi - lo) // 2; return mid on a match, else move lo = mid + 1 or hi = mid - 1; return -1. O(log n) time, O(1) space.
Use lo + (hi - lo) / 2 in Java, Go and C++ to avoid integer overflow. Python ints do not overflow, but the habit shows care.
State the loop invariant. Know the lower-bound variant (first index with a[i] >= target) using a half-open range [lo, hi) and while lo < hi. It handles duplicates and insertion points, and it is what Python's bisect_left implements.
Edge cases: empty array, a single element, target smaller or larger than every element, all duplicates. Test each.

Follow-up

Return the first and last positions of a target that appears many times.
Search a sorted array that has been rotated.
Find the minimum truck capacity that ships all packages, in order, within D days (binary search on the answer).

Solve a problem involving sorting an array with a time complexity better than O(n^2).

Medium · Coding / Algorithms

Approach

Know the options and what decides between them. Merge sort: O(n log n) always, stable, O(n) extra memory. Quicksort: O(n log n) average, O(n^2) worst case (mitigated by a random or median-of-three pivot), in place, not stable. Heapsort: O(n log n) worst case, O(1) extra space, not stable.
For bounded integer keys, counting sort runs in O(n + k) where k is the key range, and radix sort runs in O(d(n + b)) for d digits in base b. Both can beat the Omega(n log n) bound for comparison sorts when k, or d and b, are small.
Know your language's built-in: Python's sort and Java's object sort are Timsort (stable, very fast on nearly sorted data). Java sorts primitives with dual-pivot quicksort. Go's sort.Slice is not stable (use sort.SliceStable when order of equal keys matters). In C++, std::sort is introsort and not stable, while std::stable_sort is stable.
Be ready to write merge sort from scratch with tests (empty, one element, duplicates, already sorted, reverse sorted), and to apply sorting in problems like merging overlapping intervals.

Follow-up

The data does not fit in memory. How do you sort it?
When does stability matter? Give an example with a secondary sort key.
Find the k-th largest element without sorting the whole array.

Describe how you would approach a coding challenge that requires dynamic programming.

Medium · Coding / Algorithms

Approach

Recognise it: the problem asks for an optimum or a count, and a brute-force recursion recomputes the same subproblems (optimal substructure plus overlapping subproblems).
Write the brute-force recursion first, then define the state exactly (e.g. dp[i][w] = best value using the first i items with capacity w), the recurrence, the base cases and the evaluation order.
Choose top-down memoisation (closest to the recursion, computes only reachable states) or a bottom-up table (no recursion depth, easier to save space). Complexity = number of states times transitions per state.
Worked examples: coin change minimum coins in O(amount x coins); 0/1 knapsack, e.g. filling a truck's weight capacity for maximum value, in O(n x W), reducible to O(W) space with a single row iterated backwards.
Reconstruct the actual choices, not just the optimal value, by storing decisions or walking the table back.

Follow-up

How would you recover which items were chosen, not just the best value?
Reduce the memory of your table from O(n x W) to O(W).
When does a greedy algorithm give the same answer, and when does it fail?

Can you explain the concept of recursion and provide an example?

Medium · Coding / Algorithms

Approach

Definition: a function solves a problem by calling itself on a smaller instance, with a base case that stops it. Each call has a frame on the call stack until it returns.
Pick an example that is naturally recursive: the maximum depth of a binary tree is 0 for None, else 1 + max(depth(left), depth(right)). O(n) time, O(h) stack space where h is the height.
Pitfalls: a missing or unreachable base case, stack overflow on deep inputs (Python's default limit is about 1000, and neither Python nor Java optimises tail calls), and exponential repeated work, e.g. naive Fibonacci is O(2^n) and becomes O(n) with memoisation.
Show that you can convert recursion to iteration with an explicit stack (iterative DFS) when depth is unbounded.

Follow-up

Rewrite your example iteratively.
What is the time complexity of naive recursive Fibonacci, and why?
When would you choose recursion over iteration in production code?

Can you describe the trade-offs between SQL and NoSQL databases?

Medium · System Design / Architecture

Approach

Relational (PostgreSQL, MySQL): a fixed schema, joins, ACID transactions across rows and tables, and flexible ad hoc queries. It scales vertically plus read replicas, and sharding is possible but adds work.
NoSQL is several families: key-value (Redis, DynamoDB), document (MongoDB), wide-column (Cassandra) and graph. They usually scale horizontally by partition key, offer flexible schemas and tunable consistency, but have limited joins and cross-partition transactions. You model the data around known access patterns.
Decide by access patterns, consistency needs, relationship complexity and write volume, not by fashion. Orders, billing and inventory reservations fit relational transactions. High-volume GPS pings or telematics keyed by vehicle and time fit a wide-column or time-series store.
Mention the overlap: PostgreSQL JSONB handles semi-structured fields, and some NoSQL stores now offer transactions. Many systems use both, with the relational store as the source of truth.

Follow-up

At what point would you move a workload off PostgreSQL?
How do you model a one-to-many relationship in a document store, and what does it cost on update?
How do secondary indexes work in DynamoDB or Cassandra, and what do they cost?

Denormalise tenant onto revisions and backfill it live

Hard · denormalisation · online migration · backfill

resource_revision (revision_id, resource_id, version, actor_user_id, change_kind, patch, request_id, created_at) has 400M rows and no tenant column; tenant_id lives only on resource. Two reads need it: a tenant-scoped audit feed ordered by created_at DESC, and an offboarding purge. Both join back to resource today. Justify adding tenant_id to resource_revision against those two reads, name the anomaly the copy introduces and the constraint that prevents it, then give the ordered migration for a live table taking 1.2k writes/second — the lock each step takes, how the backfill is batched, and where each step stops being reversible. PostgreSQL 16.

Approach

Justify from the access path rather than from taste. Without the column, the audit feed either scans resource_revision by created_at and discards other tenants' rows, or resolves the tenant's resource_ids first and probes with them — both proportional to the tenant's whole history rather than to one page. With (tenant_id, created_at DESC, revision_id DESC) it is a seek that stops at 50 rows, and the purge becomes a ranged delete instead of a join.
Name the cost exactly: a second copy of a fact can disagree with the first. Make the disagreement unwritable rather than documented — add UNIQUE (resource_id, tenant_id) on resource so it can serve as a foreign-key target, then FOREIGN KEY (resource_id, tenant_id) REFERENCES resource (resource_id, tenant_id) on the revision table. A revision can then only ever carry its parent's tenant.
Step one, expand: ALTER TABLE resource_revision ADD COLUMN tenant_id BIGINT NULL, with no default, so it is a catalogue change and no rewrite. It still needs ACCESS EXCLUSIVE for an instant, and that instant queues behind the longest open transaction on the table while every later query queues behind it — set lock_timeout to 2s and retry rather than wait.
Step two, dual-write: deploy the writer that populates tenant_id on every new revision while reads still use the join. Reversible by redeploying the previous build, because nothing reads the column yet.

Follow-up

The backfill is half finished and a rollback is required. What state is the table in, and what does the previous build do with a half-populated column?
How do you verify the backfill actually finished, given rows are still being inserted while it runs?

What design patterns are you familiar with, and when do you apply them?

Medium · Technical / Domain Questions

Approach

Pick three patterns you have actually shipped rather than listing the Gang of Four catalogue. For each, give the problem, the pattern and its cost.
Strategy: swap an algorithm at runtime, e.g. different rate or ETA calculations per carrier. Adapter: wrap several third-party carrier APIs behind one interface so callers never see vendor formats.
Observer or publish-subscribe: shipment status changes fan out to notifications, billing and analytics without the status service knowing about them. Decorator: add retries, caching or logging around a client without changing it.
Name when you would not use a pattern: a Factory for one concrete class is indirection with no payoff, and Singleton is hidden global state that makes tests order-dependent. Dependency injection usually does the job better.
Note how language changes the answer. In Python, first-class functions often replace Strategy and Command classes. In Java, interfaces and DI frameworks make them explicit.

Follow-up

When does a Strategy pattern become worse than a simple if/else?
What problems have you seen with Singletons, and what did you replace them with?
How would you add retry and timeout behaviour to a carrier client without modifying it?

Can you explain the concept of microservices and its advantages?

Medium · Technical / Domain Questions

Approach

Define it: independently deployable services, each owning a bounded context and its own data store, communicating over the network through REST/gRPC calls or asynchronous events.
Advantages: independent deploys and release cadence per team, scaling only the hot service (e.g. tracking ingestion), fault isolation, and freedom to choose a stack per service.
Costs, stated with the same weight: network latency and partial failure, no cross-service transactions (you need sagas and eventual consistency), distributed tracing and logging, more operational overhead, and versioned contracts between teams.
Give a decision rule: a modular monolith first while domain boundaries are still moving, then split along boundaries that change independently or scale differently. Never share a database between services.
Use a logistics split as the example: orders, inventory, shipments/tracking, billing. Explain which calls stay synchronous (an availability check) and which become events (shipment delivered).

Follow-up

How would you migrate a monolith to services without a big-bang rewrite?
An order must reserve inventory and charge a customer in two different services. How do you keep them consistent?
How do you trace one slow request across five services?

Describe a challenging bug you encountered in production and how you resolved it.

Medium · Technical / Domain Questions

Approach

Structure it as a STAR story: symptom and user impact (who was affected, how badly, for how long), how it was detected (alert, customer ticket) and what made it hard (intermittent, only in production, only under load).
Spend most of the answer on the narrowing: the hypotheses you ruled out and the evidence you used (logs, metrics, traces, matching against the deploy timeline, reproducing with production-like data, git bisect).
Name the root cause precisely, e.g. a race between two workers updating the same row, timezone handling around a DST change, or connection-pool exhaustion from an unclosed client. Avoid a vague 'it was a config issue'.
Separate mitigation (rollback, feature flag, scaling out) from the real fix, then give prevention: a regression test, a new alert or dashboard, and a written postmortem.
The story must be one where you did the debugging. If a colleague found the bug, the answer does not show your skill.

Follow-up

What signal would have caught this before customers did?
How did you confirm the fix worked and was not a coincidence?
How did you communicate with stakeholders while it was ongoing?

How do you approach testing and ensuring code quality?

Medium · Technical / Domain Questions

Approach

Describe a test pyramid with real tools: many fast unit tests (pytest, JUnit, Go's testing package), integration tests against a real database in a container, contract tests between services, and a small set of end-to-end tests on key flows.
Say what you test: boundaries, error paths, idempotency and concurrency, not just the happy path. Write a regression test for every production bug before fixing it.
Design for testability: dependency injection and narrow interfaces so external carrier or payment APIs can be replaced with fakes.
Quality gates in CI: lint, formatter, type checks (mypy, or the compiler in Java/Go), and tests on every pull request. Treat coverage as a signal, not a target. Keep pull requests small so review is meaningful.
Handle flaky tests deliberately: quarantine, find the cause (time, ordering, shared state), and fix or delete. Never just retry silently.

Follow-up

How do you test code that depends on an external API you do not control?
What coverage target would you set, and why?
A test fails one run in fifty in CI. What do you do?

Explain the importance of version control in collaborative projects.

Medium · Technical / Domain Questions

Approach

Cover what version control gives a team: a full history of who changed what and why, safe parallel work on branches, review through pull requests, and the ability to revert or reproduce any release from a tag.
Compare workflows: trunk-based development with short-lived branches and feature flags (fewer painful merges, continuous integration) versus long-lived release branches in a GitFlow-style model (more ceremony, suited to versioned releases).
Know the mechanics: merge vs rebase vs squash, and why you never rewrite history on a shared branch. Write commit messages that explain why. Protect main and require CI and review before merging.
Give a concrete payoff: using git bisect to find the commit that introduced a regression, or reverting a bad merge in minutes during an incident.

Follow-up

When do you rebase and when do you merge?
How do you resolve a conflict where both sides changed the same logic?
A bad change is already deployed. Walk through rolling it back.

Design a system for real-time inventory management. What components would you include?

Medium · System Design / Architecture

Approach

Clarify scope: SKUs per warehouse location, and operations to receive, reserve (on order), pick, ship, adjust and transfer. Ask for read vs write volume, how fresh availability must be, and whether overselling is ever acceptable.
Data model: an append-only inventory ledger of quantity deltas (with reason and reference) plus a materialised balance per (sku, location) holding on_hand, reserved and available. Reserve atomically with UPDATE balance SET reserved = reserved + :n WHERE sku = :s AND loc = :l AND on_hand - reserved >= :n, and treat zero rows updated as out of stock.
Components: an API layer; the inventory service on a relational database; an event stream (e.g. Kafka) published through a transactional outbox so the order, fulfilment, analytics and dashboard consumers see every change; and a read cache or replica for availability queries, with short TTLs and the database as the source of truth on reservation.
Reliability: idempotency keys on every device and API write so scanner retries do not double-count, and reservations with an expiry so abandoned orders release stock. Run reconciliation jobs against physical cycle counts.
Scaling and trade-offs: partition by location or SKU. For hot SKUs, split counters or queue writes per key. Accept slightly stale reads on browse pages but never on checkout.

Follow-up

Two orders arrive for the last unit at the same moment. What exactly happens?
Handheld scanners go offline for an hour and then sync. How do you merge their updates?
A bug corrupted balances for a day. How do you rebuild correct state?

How would you structure a RESTful API for a logistics application?

Medium · System Design / Architecture

Approach

Model nouns as resources: /v1/shipments, /v1/shipments/{id}, /v1/shipments/{id}/events (tracking history), /v1/loads, /v1/vehicles, /v1/locations. Use HTTP verbs with their real semantics: GET safe, PUT idempotent, PATCH partial, POST create.
Status codes callers can branch on: 201 with a Location header on create, 400/422 for validation errors (with a machine-readable error body), 404, 409 for state conflicts (e.g. cancelling a delivered shipment), 429 with Retry-After.
Make POST safe to retry with an Idempotency-Key header stored with the result. Use ETag/If-Match for optimistic concurrency on updates. Model state transitions explicitly (POST /shipments/{id}/cancel) rather than letting clients PATCH the status freely.
Lists: cursor-based pagination, filters (status, date range, origin) and sorting. Long-running work (rating a large batch) returns 202 with a status URL. Push tracking updates through webhooks with signed payloads and retries instead of making customers poll.
Versioning and security: a version in the path or header, additive changes only within a version, OAuth2 scopes per customer account, and tenant checks on every resource lookup.

Follow-up

The client times out on create and retries. How do you avoid two shipments?
How would you add a required field without breaking existing integrations?
A customer polls tracking every second for 10,000 shipments. What do you change?

Discuss how you would ensure data consistency in a distributed system.

Medium · System Design / Architecture

Approach

Start by asking which data needs strong consistency (inventory reservation, payments) and where eventual consistency is acceptable (dashboards, search). Not everything needs the same guarantee.
Inside one database, use transactions and the right isolation level, or optimistic concurrency with a version column. Across services, avoid two-phase commit and use a saga: a sequence of local transactions with compensating actions (release reservation, refund).
Make events reliable with a transactional outbox: write the state change and the event row in one transaction, and have a relay publish them. Consumers must be idempotent (dedupe on event id), because delivery is at-least-once.
For replicated stores, explain quorums (R + W > N gives overlapping reads and writes) and the CAP/PACELC trade-off: during a partition you choose availability or consistency, and when there is none you trade latency for consistency.
Detect drift with periodic reconciliation jobs and alerts, because some failure will slip through.

Follow-up

The relay publishes an event and crashes before marking it sent. What happens downstream?
A compensating action fails. Now what?
How do you give a user read-your-writes consistency when reads go to replicas?

What considerations do you keep in mind when designing for high availability?

Medium · System Design / Architecture

Approach

Define the target first: an SLO such as 99.9% monthly availability (about 43 minutes of downtime a month), and RTO/RPO for data. Everything else is sized against those numbers.
Remove single points of failure: stateless services with several instances across availability zones behind a load balancer with health checks, and a replicated database with automated failover. On AWS or Azure that means multi-AZ deployments and managed failover.
Contain failures: timeouts on every call, retries with exponential backoff and jitter (only for idempotent operations), circuit breakers, bulkheads, and graceful degradation (serve cached tracking data if the live source is down).
Make change safe: rolling, blue-green or canary deploys with automated rollback, backward-compatible schema migrations, and feature flags. Many outages come from deploys rather than hardware.
Observe and rehearse: alert on SLO burn rate, keep runbooks, and test failover with game days. Weigh multi-region against its cost and consistency complexity.

Follow-up

What happens to in-flight writes during a database failover?
How do retries turn a small outage into a big one, and how do you prevent it?
Would you go multi-region for this system? What would decide it?