Amigo Software Engineer Interview Guide

The 60-second brief

Candidates describe Amigo as building an autonomous healthcare system in which AI agents carry out clinical tasks, from triage through post-visit care. The Software Engineer posting reported here is Staff level and comes in two tracks. The Infrastructure track covers Kubernetes ownership, multi-region rollout and security across the platform. The Data track covers the Databricks foundation that processes and stores agent reasoning traces and clinical outcomes so researchers can use them. Both tracks work alongside Data Scientists, Clinical Operations and Product teams.

This guide is for engineers interviewing for the Staff Software Engineer role at Amigo on either the Infrastructure or the Data track. It covers the six reported design and reliability questions, the evaluation areas candidates describe (distributed systems, security and compliance, ownership), a seven-day plan and the behavioral stories to prepare. No coding round has been reported, so confirm with your recruiter whether you will write code.

Candidates report no named round list. The described flow moves from an initial technical screen through architecture deep dives to a culture-alignment conversation, and the reported questions are all system design or infrastructure reliability questions.

Focus areas

• Databricks
• Kubernetes
• Data Engineering

Overview

The six reported questions split evenly. Three are system design: a multi-region deployment strategy with zero-downtime updates for a healthcare AI system, a data platform that combines streaming ingestion and batch processing under HIPAA, and designing for failure with disaster recovery and data isolation in a multi-tenant cloud. Three are infrastructure and reliability: auto-scaling Kubernetes clusters for unpredictable spikes without wasting money, balancing rapid development against security and compliance standards, and evaluating infrastructure-as-code tools while keeping production environments in parity. The common thread is scale, security and failure handling, not algorithms.

Candidates describe two evaluation areas. Distributed systems and scalability covers multi-region architecture, load balancing, state management, the consistency versus availability trade-off, latency in real-time data processing, multi-tenant isolation, data residency and disaster recovery testing. Security and compliance covers frameworks such as HIPAA and SOC 2. Candidates are also told to go beyond configuration into the mechanics of their tools, and to address security, latency and compliance trade-offs whenever they present a design. Choose your track early: the Infrastructure track leans on Kubernetes and cloud platforms (AWS, GCP or Azure), the Data track on Databricks and streaming pipelines.

Because the role is Staff level, the reported evaluation areas include Staff-level ownership alongside design: spotting problems before they happen, mentoring others, setting engineering standards and making build-versus-buy calls. Candidates report that interviewers challenge assumptions, and the reported tips mention skepticism and a low-ego willingness to defend a trade-off yet change course when a better argument appears. Prepare one real system from your own history that you can take apart in front of someone, including what failed.

Where to focus your preparation

Candidates describe a single progression rather than a named list of rounds, so this guide treats the loop as one preparation focus and ties it to the areas and questions candidates report.

Preparation focus

No round names or order are reported beyond a general arc: an initial technical screen, deep-dive architecture sessions, then a culture-alignment conversation, with a final round at the end. The reported questions are design and reliability questions, so the technical stages are where you rehearse them: multi-region deployment, a streaming plus batch data platform under HIPAA, disaster recovery and isolation for multi-tenant systems, Kubernetes auto-scaling, speed versus compliance, and infrastructure-as-code parity. Approach each one the same way: state requirements and constraints, sketch the architecture, walk through how each component fails, then name the trade-offs among security, latency, cost and compliance. Candidates also report that past projects are expected in STAR form with hard numbers in the result.

What to demonstrate

• System design for scale and failure: multi-region architecture, load balancing, state management, consistency versus availability, as candidates describe the distributed systems area
• Security and compliance reasoning in a healthcare setting, including familiarity with HIPAA and SOC 2
• Depth in your primary stack, Kubernetes and cloud infrastructure or Databricks and data pipelines, down to how the tools work rather than how they are configured
• Ownership at Staff level: finding problems early, mentoring, and defending or revising a design under challenge

How to prepare

• Pick your track (Infrastructure or Data) and write a one-page walk-through of the largest system you ran: components, failure modes, the trade-offs you made, and measured results such as uptime or throughput
• Rehearse each of the six reported questions aloud in one fixed order: requirements, architecture, failure modes, trade-offs, what you would measure
• Review mechanics under your tools: for Kubernetes, how the Horizontal Pod Autoscaler computes replicas and how node scaling reacts to pending pods; for Databricks, how streaming checkpoints and the Delta transaction log make pipelines restartable and idempotent
• Ask your recruiter for the stage sequence, whether you will write code, and which track the loop targets

Where candidates lose points

Answering the multi-region question with 'deploy to two regions behind a load balancer'

Cover the parts that make it hard: how traffic is steered and drained on failure, how data is replicated and what RPO and RTO that gives, and how schema changes roll out with backward-compatible steps so a release never needs downtime. Say whether you run active-active or active-passive and why.

Treating HIPAA or SOC 2 as a checkbox in the data-platform and speed-versus-compliance answers

Name concrete controls: encryption in transit and at rest, least-privilege access with audit logs, separation of identifiable and de-identified data, retention and deletion rules, and automated policy checks in the delivery pipeline so compliance does not slow every release. Then say what it costs in latency or complexity.

Answering the Kubernetes auto-scaling question with 'turn on the HPA'

Explain the whole chain: which metric drives pod scaling (CPU, queue depth or request rate), how resource requests affect it, how long new nodes take to arrive, how spare headroom or low-priority buffer pods absorb a sudden spike, and how you cap cost with node-pool mix and scale-down rules. Include disruption budgets so scale-down does not cause an outage.

Designing for the happy path in the multi-tenant disaster recovery question

List failure modes first: region loss, a bad deploy, a corrupted table, one tenant overloading shared resources. Pair each with a mitigation (cross-region backups, per-tenant quotas, tenant-scoped encryption keys) and say how you test it with restore drills and failure exercises, since an untested backup is only a hope.

Telling project stories with no numbers, no failure, or no willingness to move

Prepare STAR stories whose result carries a measured figure (uptime, latency, cost, incident count), one story about your biggest professional failure and what you changed afterwards, and one where you defended a design under challenge and either held it with evidence or changed it when a better argument came up.

Questions & practice

Merge partitioned event streams into one ordered feed with bounded lateness

Hard · k-way merge · watermarks · out-of-order streams

The read-model service consumes 64 log partitions carrying about 4,000 events per second in total. Each partition is ordered within itself, but partitions drift by up to 30 seconds, and the activity feed must present a tenant's events in occurred_at order. Produce the merge. State its complexity, the buffer it requires in events and in bytes, what happens when one partition is idle, and what you do with an event that arrives after you have already emitted its position. Payloads average 1 KB.

Approach

1. Merge with a min-heap over the 64 partition heads keyed on (occurred_at, event_id): O(log P) per event and O(n log P) overall. The tie-break on event_id is what makes the output deterministic when two partitions carry the same millisecond, which matters because the feed is paginated and a non-deterministic order reorders pages under the reader.
2. Emitting the heap head is only correct once every partition has produced everything up to that timestamp, so the emit condition is a watermark: the minimum across partitions of the highest occurred_at seen, less the allowed lateness. Events are held until the watermark passes them, which is what turns individually ordered streams into a jointly ordered one.
3. Size the buffer from the lateness rather than guessing: 4,000 events per second times 30 seconds is 120,000 buffered events, and at 1 KB each about 120 MB of heap. That number is the real price of the ordering guarantee and belongs in front of whoever asked for it.
4. Handle the idle partition explicitly, because it fails the feed rather than corrupting it: a partition with no traffic never advances its own maximum, so the watermark freezes and output stops entirely. Either every partition emits a periodic idle marker carrying the broker's current time, or the watermark falls back to wall clock for a partition silent beyond a threshold.

Follow-up

• The lateness budget is raised to five minutes. What is the new buffer, and what besides memory changes?
• The consumer restarts. Where does it resume from, and what does the feed look like for the first 30 seconds?

Archive a resource graph without breaking live references or recursing

Medium · graph traversal · topological order · tenant isolation

Resources reference other resources within a tenant; for the largest tenant the reference table holds up to 2,000,000 nodes and 8,000,000 edges. Archiving a resource must archive everything reachable from it that nothing outside the set still references, refuse when a live external referrer exists, and terminate when references form cycles, which they legitimately do. Produce the archive order and the refusal list, targeting O(V+E). Say what stops the traversal crossing a tenant boundary, and why recursion is the wrong control structure at this size.

Approach

1. Load the subgraph with the tenant predicate on both endpoints of the edge, not only on the side you started from. Scoping the left table alone is the classic cross-tenant leak: one mis-entered edge then pulls another tenant's resources into the traversal and, worse, into the archive.
2. Traverse iteratively with an explicit stack. A 2,000,000-node graph can hold a chain deep enough to exhaust a native stack in the low tens of thousands of frames, and that failure is a process crash rather than an error you can return.
3. Treat cycles as data rather than corruption: compute strongly connected components with Tarjan in O(V+E) using its own explicit stack, then condense. The condensation is a DAG, so a topological order over it gives the archive order, and every member of a component archives in one transaction because no order within a cycle is valid.
4. Decide refusals with reverse edges. A candidate is archivable only if every in-edge originates inside the candidate set, so build the transpose or count in-degrees restricted to the visited set, and emit each blocked resource with the id of the external referrer, which is the only part of the answer an operator can act on.

Follow-up

• The graph is read in one query and the archive writes a minute later. What can change in between, and how do you make the write safe?
• The candidate set is 400,000 resources. Is that one transaction, and if not, what does a half-finished archive look like to a reader?

Collapse a redelivered event batch into per-aggregate high-water marks

Easy · hashing · at-least-once · aggregation

You drain a batch of up to 5,000,000 events, each (aggregate_id BIGINT, aggregate_version INT, event_type, payload). The log guarantees order within one aggregate only; the batch merges 64 partitions, and a relay failover has redelivered a range, so an older version for an aggregate can appear after a newer one. Given a map of last_applied_version per aggregate, produce the events worth applying, at most one per (aggregate_id, version), plus the count discarded. Target O(n) time. State the memory for 2,000,000 distinct aggregates and what you do when it does not fit.

Approach

1. One pass, one hash map from aggregate_id to the highest version kept, and a discard counter. An event whose version is at or below last_applied_version for its aggregate is dropped without further work, which is the whole reason the event carries its version rather than a delta. O(n) expected time, O(d) space in distinct aggregates.
2. Keep the maximum, never the last occurrence. The redelivered range means the final appearance of an aggregate in the batch can be an older version than one seen earlier in the same batch, so last-wins applies stale state over newer state and the projection regresses with no error anywhere.
3. Cost the memory instead of calling it large: an 8-byte key plus a 4-byte version is 12 bytes of payload, and an open-addressed table held at a 0.7 load factor costs roughly 17 bytes per entry before per-slot metadata, so 2,000,000 aggregates is tens of megabytes in a native layout and several times that in a runtime that boxes both key and value.
4. If the distinct set exceeds memory, partition on hash(aggregate_id) mod P and reduce each partition independently. Every event for one aggregate hashes to the same partition, so the per-partition result is exact and the merge is concatenation rather than a second reduction.

Follow-up

• The payload is a patch rather than a snapshot, so applying only the highest version loses the intermediate changes. What changes in your reduction?
• How do you detect that version 7 arrived while version 6 was never delivered, and what should the consumer do about the gap?

Denormalise tenant onto revisions and backfill it live

Hard · denormalisation · online migration · backfill

resource_revision (revision_id, resource_id, version, actor_user_id, change_kind, patch, request_id, created_at) has 400M rows and no tenant column; tenant_id lives only on resource. Two reads need it: a tenant-scoped audit feed ordered by created_at DESC, and an offboarding purge. Both join back to resource today. Justify adding tenant_id to resource_revision against those two reads, name the anomaly the copy introduces and the constraint that prevents it, then give the ordered migration for a live table taking 1.2k writes/second — the lock each step takes, how the backfill is batched, and where each step stops being reversible. PostgreSQL 16.

Approach

1. Justify from the access path rather than from taste. Without the column, the audit feed either scans resource_revision by created_at and discards other tenants' rows, or resolves the tenant's resource_ids first and probes with them — both proportional to the tenant's whole history rather than to one page. With (tenant_id, created_at DESC, revision_id DESC) it is a seek that stops at 50 rows, and the purge becomes a ranged delete instead of a join.
2. Name the cost exactly: a second copy of a fact can disagree with the first. Make the disagreement unwritable rather than documented — add UNIQUE (resource_id, tenant_id) on resource so it can serve as a foreign-key target, then FOREIGN KEY (resource_id, tenant_id) REFERENCES resource (resource_id, tenant_id) on the revision table. A revision can then only ever carry its parent's tenant.
3. Step one, expand: ALTER TABLE resource_revision ADD COLUMN tenant_id BIGINT NULL, with no default, so it is a catalogue change and no rewrite. It still needs ACCESS EXCLUSIVE for an instant, and that instant queues behind the longest open transaction on the table while every later query queues behind it — set lock_timeout to 2s and retry rather than wait.
4. Step two, dual-write: deploy the writer that populates tenant_id on every new revision while reads still use the join. Reversible by redeploying the previous build, because nothing reads the column yet.

Follow-up

• The backfill is half finished and a rollback is required. What state is the table in, and what does the previous build do with a half-populated column?
• How do you verify the backfill actually finished, given rows are still being inserted while it runs?

Explain why the owner filter ignores the listing index

Medium · composite index · query plans · sort elimination

The only index on resource is (tenant_id, status, updated_at DESC, resource_id DESC). A new endpoint returns one user's resources across all statuses, newest created first: WHERE tenant_id = $1 AND owner_user_id = $2 ORDER BY created_at DESC LIMIT 20. On a tenant with 2M rows it takes 900 ms and EXPLAIN shows a sort above a large scan. Explain precisely why the existing index cannot serve it, give the index that can, and state which of these the new index still will not help: owner_user_id alone across tenants; the same query ordered by updated_at. PostgreSQL 16.

Approach

1. Separate the two jobs an index does. For filtering, a composite btree is seekable only on a left prefix, so with no predicate on status the scan can at best range over tenant_id and test owner_user_id per row; PostgreSQL 16 has no btree skip scan to jump the unconstrained column.
2. For ordering, the index is sorted by (status, updated_at) within a tenant and not by created_at, so the LIMIT cannot stop early: every matching row is read and then sorted. That is the 'Sort Method: top-N heapsort' line, and it is why the plan reads 2M rows to answer with 20.
3. Derive the replacement from the access path — equality, equality, then the ordering column: CREATE INDEX CONCURRENTLY ON resource (tenant_id, owner_user_id, created_at DESC). The scan seeks to the (tenant, owner) range and walks 20 entries in order, so the Sort node disappears along with the row-read.
4. Treat INCLUDE (title, status) as conditional, not free. An index-only scan still visits the heap for any row whose page is not marked all-visible, so on a table taking 1.2k writes/second the win depends on autovacuum keeping the visibility map current, and the wider index costs more on every insert.

Follow-up

• 90% of rows are status='active'. Would a partial index WHERE status = 'active' change your answer, and for which of the three queries?
• A dashboard runs this for 40 owners in one page load. What changes about the design?

How would you design a multi-region deployment strategy for a high-availability healthcare AI system that requ

Medium · System Design & Architecture

How would you design a multi-region deployment strategy for a high-availability healthcare AI system that requires zero-downtime updates?

Approach

1. Pin the targets first: RTO and RPO per service, whether patient data must stay inside a region, and whether active-active or active-passive is acceptable. Stateless services and model servers run in every region behind a health-checked global load balancer (DNS-based or anycast); state is the hard part.
2. Choose the data strategy openly: one write region with asynchronous replication gives low write latency but a non-zero RPO on failover; a synchronous or consensus-replicated multi-region database gives RPO near zero but adds a cross-region round trip to every commit. Prevent split brain with a single-writer lease or fencing token.
3. Zero-downtime rollout on Kubernetes: rolling updates with maxUnavailable 0 and maxSurge above 0, readiness probes so new pods take traffic only when ready, PodDisruptionBudgets, and a preStop hook plus termination grace period so in-flight requests and long-lived connections drain.
4. Release progressively: canary one region (or one cell) first, compare error rate and latency against the old version, and roll back automatically on an SLO breach. For model changes, run shadow traffic and pin a session to one model version so a single conversation never switches mid-stream.
5. Keep two versions compatible at once: expand/contract schema migrations (add, dual-write, backfill, switch reads, remove), backward-compatible APIs, and rehearsed regional failover. Remember that DNS failover is bounded by record TTL and client caching.

Follow-up

• A schema change ships while old and new pods both serve traffic. Walk through each step so neither version breaks.
• During a regional failover, what happens to in-flight writes and active sessions, and how do you stop both regions from accepting writes?
• How would you roll out a new model version so that a regression shows up before every region gets it?

Describe your approach to architecting a data platform that handles streaming ingestion and batch processing w

Medium · System Design & Architecture

Describe your approach to architecting a data platform that handles streaming ingestion and batch processing while ensuring strict HIPAA compliance.

Approach

1. Start from the data this platform holds: agent reasoning traces and clinical outcomes, which are high-volume, semi-structured and likely to carry PHI. Land every event in a durable, replayable log (Kafka or Kinesis) and tokenize or strip direct identifiers at ingest, before the bronze layer. Write raw tables to object storage, then build silver and gold Delta tables on Databricks. Key deduplication on the encounter or session id plus an event id.
2. Use one streaming path: Spark Structured Streaming serves the low-latency tables, and a backfill replays the log through that same streaming job (kappa-style) rather than keeping separate batch logic. Name the cost: replaying a large history through the streaming path is slow, so reprocessing runs need their own capacity.
3. Make correctness explicit: checkpoints plus idempotent sinks (Delta MERGE on a natural or event key) for effectively-once results, event-time watermarks for late data, and a schema registry or contract checks so a producer change cannot silently corrupt tables.
4. HIPAA controls: a BAA with every vendor that touches PHI, TLS in transit, KMS-managed encryption at rest, least-privilege access with row- and column-level controls (for example Unity Catalog), and audit logs of who read PHI. Apply minimum necessary by de-identifying early (Safe Harbor or Expert Determination) and keep the re-identification map in a separate, tightly restricted store.
5. Cover retention and lineage: Delta time travel keeps old files until VACUUM runs (default retention is 7 days), so corrections and retention rules must account for old versions. Track lineage from raw to gold so you can tell which outputs a bad record reached.

Follow-up

• A transform bug corrupted a month of silver data. How do you reprocess it without double-counting or breaking downstream consumers?
• How do data scientists get research access to clinical outcomes without seeing direct identifiers?
• An upstream producer adds a field and changes a type without warning. What catches it and what happens to the stream?

How do you design for failure? Walk us through your strategy for disaster recovery and data isolation in a mul

Medium · System Design & Architecture

How do you design for failure? Walk us through your strategy for disaster recovery and data isolation in a multi-tenant cloud environment.

Approach

1. Name the failure domains (process, node, availability zone, region, cloud account) and set an RTO/RPO for each data tier. Then pick a DR pattern per tier: backup-and-restore, pilot light, warm standby or active-active, each trading cost against recovery time.
2. Contain failures in the request path: timeouts on every call, retries with exponential backoff and jitter only on idempotent operations (or with idempotency keys), circuit breakers, bulkheads, and graceful degradation. A cell-based layout limits the blast radius to one group of tenants.
3. Backups: point-in-time recovery, copies in another region and another account, and immutable (object-lock) storage so ransomware or a compromised admin cannot delete them. Run regular restore drills and measure the actual RTO; a backup that has never been restored is untested.
4. Tenant isolation is a spectrum: silo (database or account per tenant), bridge (schema per tenant) or pool (shared tables keyed by tenant_id with row-level security, such as Postgres RLS). Set tenant context from the authenticated identity, not from request parameters, and consider per-tenant encryption keys so destroying a key makes that tenant's data unreadable.
5. Connect DR and isolation: in a pooled database, restoring one tenant means a logical restore into a side copy and a filtered merge back, not a whole-cluster rollback. Add per-tenant quotas against noisy neighbours, and run game days that fail a zone or dependency on purpose.

Follow-up

• One tenant needs its data back as of yesterday in a shared database. How do you do it without touching other tenants?
• How would you show that a code bug cannot return one tenant's records to another tenant?
• The standby region has never served production traffic. How do you gain confidence that failover works?

What specific strategies do you use for auto-scaling Kubernetes clusters to handle unpredictable traffic spike

Medium · Infrastructure & Reliability

What specific strategies do you use for auto-scaling Kubernetes clusters to handle unpredictable traffic spikes while maintaining cost efficiency?

Approach

1. Scale in layers. The Horizontal Pod Autoscaler sets replicas as ceil(current replicas x current metric / target metric). Use custom or external metrics (requests per second, queue depth, in-flight requests) when CPU lags the real load. Use the Vertical Pod Autoscaler or its recommendations to right-size requests, but never let both autoscalers act on the same CPU or memory metric.
2. Add nodes with Cluster Autoscaler or Karpenter, both of which react to unschedulable pods. Node startup takes much longer than pod startup, so keep headroom with low-priority overprovisioning pods that real workloads preempt, pre-pull large images, and use scheduled scaling for known peaks. KEDA covers event-driven scaling and scale-to-zero.
3. Tune HPA behavior: aggressive scale-up policies and a scale-down stabilization window (default 300 seconds) so replicas do not flap after a spike. Inference services with slow model loading need extra headroom, because a new replica is useless until the model is in memory.
4. Control cost: the scheduler packs nodes by requests, not actual usage, so accurate requests matter most. Run interruption-tolerant stateless work on spot or preemptible nodes with an on-demand baseline and several instance types, enable consolidation, and set PodDisruptionBudgets so consolidation never takes out a service's minimum replicas.
5. Protect what scaling cannot fix: cap database connections (pooling, or a pooler such as PgBouncer) so scale-out does not exhaust them, and add rate limiting and load shedding for spikes that outrun provisioning.

Follow-up

• CPU-based scaling reacts too late for a latency-sensitive inference service. Which metric would you scale on, and how do you feed it to the HPA?
• Spot nodes are reclaimed in the middle of a traffic spike. What keeps the service within its SLO?
• After a scale-out, the database runs out of connections. How do you diagnose it and stop it happening again?

How do you balance the need for rapid development with the "non-negotiable" security and compliance standards

Medium · Infrastructure & Reliability

How do you balance the need for rapid development with the "non-negotiable" security and compliance standards required in healthcare?

Approach

1. Make the compliant path the easiest path: shared templates and infrastructure modules with encryption, logging, network policies and least-privilege IAM built in, so a new service is compliant without extra work. Guardrails beat manual approval gates.
2. Automate the checks in CI and at admission: policy as code (OPA Gatekeeper or Kyverno in the cluster, Checkov or conftest on Terraform plans), dependency and container image scanning, secret scanning and signed images. Developers then get feedback on the pull request, not weeks later in an audit.
3. Separate data by sensitivity: synthetic or de-identified data in development so engineers work without PHI access; production PHI access only just-in-time, time-boxed and audited, with a documented break-glass path for incidents.
4. Make review depth match the risk: lightweight review for low-risk changes, threat modelling and security sign-off for anything touching PHI, authentication or tenant boundaries. Collect audit evidence (for SOC 2 or HIPAA) automatically from the pipeline.
5. Tell it as a STAR story with numbers: a guardrail that cut lead time or review load, and a shortcut you refused, with the compensating control you offered instead. Keep a clear set of non-waivable controls, such as PHI encryption and audit logging. The answer fails if it treats security as a pure blocker or claims that no control can ever be relaxed under any circumstances.

Follow-up

• An engineer needs to query production PHI tonight to debug an incident. What is the access path, and what record does it leave?
• Which control would you never waive under deadline pressure, and which would you relax temporarily with a compensating control?
• How would you measure whether your guardrails are slowing engineers down?

Explain your process for evaluating infrastructure-as-code tools and how you ensure parity across multiple pro

Medium · Infrastructure & Reliability

Explain your process for evaluating infrastructure-as-code tools and how you ensure parity across multiple production environments.

Approach

1. Evaluate against explicit criteria: state handling (Terraform/OpenTofu remote state with locking, Pulumi's state backend, or provider-managed state in CloudFormation), language (HCL or general-purpose languages in Pulumi/CDK), provider coverage across the clouds you run, plan/preview quality, drift detection, testing support, licensing (Terraform moved to the BSL in 2023, which led to the OpenTofu fork) and team skills.
2. Run a bake-off on a real slice of infrastructure, such as one VPC, cluster and database, instead of comparing feature lists. Count the migration cost: importing existing resources (import blocks or terraform import) without recreating them.
3. For parity, use one set of versioned modules for every environment, with differences only in explicit variables (region, sizes, replica counts). Promote the same module version from staging to each production environment, and pin provider versions with the dependency lock file.
4. Detect and prevent drift: run a scheduled plan in CI that alerts on any non-empty diff, restrict console write access through IAM, and manage Kubernetes manifests with GitOps (Argo CD or Flux, using Kustomize overlays or Helm values per environment) so the cluster reconciles to what is in git.
5. Isolate state per environment and region to limit the blast radius of a bad apply. Document the differences you allow on purpose (data residency, regional capacity) so they are not mistaken for drift.

Follow-up

• Someone changed a security group in the console during an incident. How do you detect it and bring code and reality back in line?
• How do you roll a module change across many production environments without applying it everywhere at once?
• How would you move hand-built resources under code management without downtime or recreation?

One log partition stops advancing while the others drain

Easy · poison message · head-of-line blocking · consumer lag

Search results for a subset of tenants are hours stale; the rest are current. The projection consumer reports lag of zero on 15 of 16 partitions and 400,000 on one. Its error rate is flat and its CPU is idle. outbox_event has no pending rows older than a second, so the relay has published everything it holds. Identify the mechanism, give the ordered checks, and state what you do in the first ten minutes versus what you change permanently.

Approach

1. Read the lag distribution first. A slow consumer lags everywhere; zero on fifteen partitions and 400,000 on one is not throughput. Idle CPU on the stuck partition means the consumer is not advancing its offset at all, which points at one message it cannot get past rather than at a rate problem.
2. Exonerate the producer before touching the consumer. No pending outbox rows older than a second means the relay published, so the event exists in the log. This separates never sent from sent and never applied, which are different code paths and usually different owners.
3. Read the message at the stuck offset and the handler's log lines for its event_id. A flat error rate with no progress has two explanations and you must distinguish them: the handler is throwing and the retry loop is swallowing it, or the handler is blocking on something and never returning. Idle CPU with no error lines favours the second.
4. Mitigate before diagnosing further. Move the offending event to a dead-letter store and commit the offset past it. Adding consumers does nothing here, because a partition is consumed by exactly one member of the group, and the blast radius is every aggregate hashed to that partition, not only the aggregate that produced the bad event.

Follow-up

• The dead-lettered event carried aggregate_version 7 and the projection had applied 6. What must the replay do differently if 8 and 9 landed in the meantime?
• How do you show staleness to the user while the partition is behind, given the API already returns the projection's watermark?

Suggested practice checklist

Seven days built around the six reported design and reliability questions, the Kubernetes and Databricks stack, and the Staff-level ownership stories. Each day ends with something written or spoken that you can review.

Day 1 — Fix your track and inventory your own systems

• Ask your recruiter which track the loop targets (Infrastructure or Data), what stages to expect and whether you will write code.
• List the three largest systems you have owned and, for each, one outage or near-miss, the design decision behind it and a measured outcome.
• Write the one-page walk-through of your strongest system: components, data flow, failure modes, trade-offs among security, latency and cost.

Deliverable: A recruiter reply with the format, a ranked list of three systems with a measured outcome each, and the one-page system walk-through.

Day 2 — Multi-region deployment with zero-downtime updates

• Sketch active-active and active-passive layouts for a healthcare AI service, marking traffic steering, data replication and the failover trigger; state RPO and RTO for each.
• Write the release sequence for a schema change that must not cause downtime: add the new column, dual-write, backfill, switch reads, then remove the old column.
• Describe a canary rollout per region with the metrics that stop it, and how you drain connections during a deploy.
• Answer the multi-region question aloud and record it.

Deliverable: Two annotated architecture sketches, a written release sequence for a zero-downtime schema change, and one recorded spoken answer.

Day 3 — Streaming plus batch data platform under HIPAA

• Draw one pipeline in which streaming ingestion and batch reprocessing share the same storage layers, with layers for raw, cleaned and curated data; mark where identifiable data is isolated.
• Write down how you get restartable, idempotent processing: checkpoints, deduplication keys, handling of late events and replay. If you use Databricks, map this to Structured Streaming and Delta tables.
• List the compliance controls the platform needs (encryption, access control, audit logging, retention and deletion, lineage) and where each is enforced.

Deliverable: One pipeline diagram with compliance controls marked, a written idempotency and replay checklist, and a completed practice exercise.

Day 4 — Designing for failure: disaster recovery and tenant isolation

• Build a failure table for a multi-tenant platform: region loss, bad deploy, data corruption, noisy tenant, credential leak. For each, write the detection signal, the mitigation and the recovery step.
• Compare tenant isolation by row-level filtering, by schema and by separate database or account, in terms of blast radius, cost and operational effort; choose one and justify it.
• Define backup and restore targets and write a restore-drill script outline; note data residency constraints per region.

Deliverable: A completed failure table, an isolation comparison with a recommendation, a restore-drill outline and a worked tenant-backfill exercise.

Day 5 — Kubernetes auto-scaling and infrastructure-as-code parity

• Write how the Horizontal Pod Autoscaler computes desired replicas (current replicas times current metric over target), which metric you would scale on for a spiky workload, and why resource requests matter.
• Describe how node scaling responds to pending pods, how spare capacity or low-priority buffer pods hide node start-up delay, and where you use on-demand versus cheaper interruptible nodes; add disruption budgets and scale-down limits.
• Write your evaluation criteria for infrastructure-as-code tools (state handling, preview of changes, drift detection, module reuse, policy checks, team skills) and a promotion pipeline that keeps environments identical except for declared variables.
• Answer both questions aloud: auto-scaling and IaC evaluation.

Deliverable: A one-page autoscaling design for a spiky workload, an IaC evaluation scorecard with a promotion pipeline outline, and two recorded answers.

Day 6 — Speed versus compliance, and the ownership stories

• Write your answer to the speed-versus-compliance question around guardrails: policy checks in the pipeline, pre-approved templates, risk-based review, automated evidence for audits, and a logged emergency-access path.
• Write four STAR stories with measured results: your biggest professional failure, a problem you caught before it happened, a time you mentored someone to a higher bar, and a build-versus-buy decision.
• Write a two-sentence answer to why you want to apply your engineering skills to healthcare, grounded in something you have actually built or lived.
• Rehearse one story where an interviewer challenges your design: what you defend with evidence, and what would make you change it.

Deliverable: A written compliance-guardrails answer, four STAR stories each with a figure, a two-sentence healthcare motivation, and a practised challenge response.

Day 7 — Full mock and gap check

• Run a spoken mock with all six reported questions in a row, drawing diagrams as you go, and record it.
• Replay the recording and mark every spot where you named a tool without explaining its mechanism, or gave a design with no failure case.
• Write two questions to ask your interviewers about the platform and the team.

Deliverable: One recorded mock, a list of marked gaps with the reworked answers, a debugging walkthrough and two questions to ask.

Behavioural preparation

Reported preparation tips mention talking about your biggest professional failure, explaining why you want to apply your engineering to healthcare, and defending a design while staying willing to change it. The reported Staff-level expectations mention finding problems before they occur and mentoring others. Candidates are also advised to use the STAR structure and end each story with hard figures such as scale and uptime change.

1. Tell me about your biggest professional failure. What did you miss, what did it cost, and what changed in how you work afterwards?
2. Why Amigo, and why healthcare? Connect your answer to something specific you have built or cared about.
3. Describe a time someone challenged a design of yours. What did you defend with evidence, and what made you change your mind, if anything did?
4. Tell me about a problem you spotted before it became an incident. How did you notice it, who did you convince, and what was the measured result?
5. Describe how you have mentored engineers to raise the technical bar on a team. What changed, and how do you know?
6. Tell me about a time you had to choose between shipping quickly and meeting a security or compliance requirement. What did you decide, and what did you put in place to keep the risk bounded?

Turn a code review disagreement into a decision

Easy · code review · optimistic concurrency · communication

A colleague's change updates a row with UPDATE resource SET version = version + 1 WHERE resource_id = $1 AND version = $2 and treats an affected-row count of zero as a successful no-op. You read that as a silently lost update; they think returning 200 is friendlier to clients than returning a conflict. Describe how you have handled a review disagreement of this shape: what goes in the comment, when you leave the thread, and who decides. Then write the comment you would leave here, in under 80 words.

Approach

1. Sort the disagreement before writing anything. A silently discarded write is a correctness claim about data; the choice between 409 and 412 is taste. Only the first justifies blocking a merge, and saying which one you are doing is most of the value of the comment.
2. Make the claim reproducible in the comment itself with an interleaving rather than a principle: A reads version 7, B reads version 7, B commits version 8, A's predicate matches zero rows, A is told it succeeded and A's edit is gone.
3. Offer the alternative with its cost attached: return 409 carrying the current version and the revision that won, so the client can re-read and re-apply. Note that automatic retry is not the fix, because a retry re-reads the winner's state and reapplies an intent formed against data that no longer exists.
4. Apply an escalation rule you can state: two round trips on the thread, then a call, and the service's owner decides rather than the reviewer. A reviewer who cannot be overruled is a bottleneck with extra steps.

Follow-up

• Where would you put the test that fails if someone reintroduces the swallowed zero rowcount?
• The author says clients cannot handle a 409. How do you check whether that is true?

Argue against a design, lose, and commit anyway

Medium · disagreement · service boundaries · decision records

Describe a design you argued against and lost. State the failure you predicted as a named mechanism, not a feeling about complexity: two services that would need one transaction, a projection with no rebuild path, a write path with no idempotency key. Say what evidence you brought, what the decision maker weighed instead, and what you did after the decision was made: what you instrumented, what you wrote down, and whether the prediction came true. Five minutes.

Approach

1. State the prediction in falsifiable form up front: the mechanism, the condition that triggers it, and the observable outcome. A prediction that cannot be checked also cannot be credited to you later.
2. Show the evidence you had at the time and label each piece honestly as measured, analogous, or intuition. Keeping the intuition is fine; disguising it as data is the thing that erodes your standing in the next argument.
3. Represent the opposing case at full strength, including the constraint you did not control: a fixed date, a team boundary, or the fact that the decision was cheap to reverse and yours was not.
4. Make disagree-and-commit concrete. Name the artefact you left behind so the prediction could be settled without you: the alert and its threshold, the counter on the dashboard, the decision note that recorded the trade-off and the condition that would revisit it.

Follow-up

• What threshold on that alert would have proved you right, and did anyone ever look at it?
• If the same proposal arrived tomorrow with the same deadline, would you argue it the same way?

Ship under a deadline and bound the debt you chose

Medium · pagination · technical debt · tradeoffs

You have four days to ship a tenant-facing listing endpoint. The version you would defend uses keyset pagination over (tenant_id, status, updated_at DESC, resource_id DESC); the version you can finish uses LIMIT/OFFSET with no matching index. Describe a deadline call you actually made of this shape: what you shipped, what you knowingly deferred, how you bounded the damage with a mechanism rather than an intention, and the specific numeric condition that would force the follow-up. Name who you told and where you wrote it down.

Approach

1. Name the deferred failure precisely instead of calling it slow. OFFSET n makes the database produce and discard n rows, so cost grows with page depth; without an index matching the sort, every matching row is read and sorted before the limit applies; and rows inserted between two page fetches shift across the boundary so items are skipped or repeated with nothing in the response to signal it.
2. Bound the blast radius with something mechanical rather than a promise: cap maximum page depth, cap page size, restrict the endpoint to one internal caller, or keep it behind a flag. State which failure each cap removes and which it leaves standing.
3. Attach a number to the trigger and wire it to an alarm: the first tenant crossing N resources, or the endpoint's p99 crossing its share of the 400 ms budget, so the debt announces itself instead of waiting to be remembered.
4. Write it where the next engineer looks, which is the code and the ticket, not a chat message: what was deferred, why, the cap, and the trigger.

Follow-up

• At what page depth does the offset version breach your latency budget, given your page size and row counts?
• What breaks first when you switch to keyset pagination later, and what does a client holding an old page token see?