Articles
Fault Injection Interview Questions: Hypotheses, Blast Radius, and Recovery Evidence
Answer fault injection interview questions with a concrete hypothesis, bounded blast radius, stop conditions, and recovery evidence.
A strong fault-injection interview answer is not, “I would kill a server and see what happens.” It states a reliability claim that can be disproved, introduces the smallest relevant fault, defines who can stop the experiment, and collects evidence that the system recovered without corrupting work.
Fault injection deliberately creates a disruptive condition so a team can observe a system's behavior. The hard part is designing an experiment whose result changes a decision. An interviewer is listening for the difference between an observable hypothesis and a vague confidence ritual, a narrow blast radius and an uncontrolled incident, and a recovered HTTP 200 and a healthy business workflow.
Evidence boundary: Official facts in this article are limited to linked AWS Fault Injection Service (AWS FIS) and Google SRE material checked in September 2026. AWS FIS documents experiment templates with actions, targets, and stop conditions, and warns that its actions affect real resources. Candidate-reported evidence is not used: no claim is made about an employer's exact interview loop or tooling. Inference covers the checkout example, thresholds, and preparation advice; treat those as a way to reason, not a production runbook. Start by practicing the adjacent judgment in Evaluate defensive testing and pair programming.
What is the interview-ready definition of fault injection?
Fault injection is a controlled experiment that introduces a defined failure mode—such as added dependency latency, an unavailable replica, dropped packets, exhausted connection capacity, or a malformed response—to test a stated expectation about system behavior. The test is not successful because the fault happened. It is useful when the team can compare a predicted steady state and failure response with measured results.
AWS FIS provides useful vocabulary: its experiment template contains actions, targets, and stop conditions. An action is the disruption, a target identifies what is exposed, and a stop condition ends the test when a guardrail is crossed. Add two pieces: the hypothesis before the action, and recovery evidence after it.
Avoid saying fault injection proves a system is resilient. One experiment covers one fault model, target selection, workload, and time window. It can increase or decrease confidence in a claim. It cannot establish that every dependency failure, regional outage, data bug, or human response will be safe.
Start with a hypothesis that could fail
Use a sentence with four parts: fault, scope, expected user-visible behavior, and measurable bound. For example:
If the inventory service adds 800 ms to 5% of reads for one canary checkout shard, then checkout will return a cached availability result or a clear retryable response; error rate stays below 1%, no duplicate orders appear, and the affected shard's request queue returns to baseline within five minutes after the fault stops.
This is better than “the service should handle latency.” It says what is delayed, where it happens, what graceful behavior is acceptable, and which measurements could disprove the claim. It also exposes decisions an interviewer can challenge: Is cached availability safe for this product? Is 1% an agreed limit? What prevents an 800 ms delay from consuming all worker threads?
Write the steady state before the test. AWS describes stop conditions in terms of a defined steady state and a CloudWatch alarm threshold; an alarm has no meaning without an acceptable baseline. Record normal success rate, latency distribution, queue age, saturation, and data-integrity signals. A green average latency can coexist with a rising retry queue or a payment reconciliation mismatch.
| Weak statement | Testable replacement | Why it changes the experiment |
|---|---|---|
| “The cache makes us reliable.” | “If the primary read path is unavailable for 60 seconds, the canary serves values no older than the product limit and records cache age.” | Names a failure, a scope, and a correctness boundary |
| “Retries will recover.” | “A transient dependency timeout causes at most two idempotent attempts per logical order; retry traffic stays within budget.” | Tests amplification as well as availability |
| “Alerts will fire.” | “The alert reaches the named on-call route within three minutes and the recovery dashboard distinguishes timeout from rejection.” | Turns observability into a measurable outcome |
How do you choose a safe blast radius?
Blast radius is the users, requests, resources, and downstream effects that can be touched if the experiment behaves worse than predicted. A 1% experiment can be unsafe if it includes a shared database writer or every request from a high-value tenant.
Begin in a disposable or pre-production environment with a representative workload. AWS explicitly recommends planning and pre-production experiments before production because the service performs real actions on real resources. An interviewer should hear that you know approval, ownership, and rollback are prerequisites, not an afterthought.
Shrink the production candidate by dimensions that matter:
- Target selection: one tagged canary instance, one noncritical partition, or an isolated test tenant before a fleet or region.
- Traffic exposure: a small, observable request slice with no silent spillover to shared queues.
- Fault intensity and duration: 200 ms before 800 ms; 30 seconds before five minutes, provided the shorter experiment can answer the hypothesis.
- Business criticality: exclude irreversible writes, payroll cutoff jobs, or a peak sales period unless the hypothesis and safeguards specifically cover them.
- Time to human control: name the operator and make a stop action available before the fault begins.
Do not confuse a circuit breaker with a blast-radius plan. A breaker can limit calls after it opens, but it may be configured incorrectly, trip too late, or push traffic toward another dependency. Test the containment mechanism, its fallback result, and its effect on callers.
Which guardrails stop the experiment?
Choose a small set of signals that represent harm, not every graph in the dashboard. AWS FIS stop conditions can be based on CloudWatch alarms, and a triggered condition stops the experiment; a stopped experiment cannot simply be resumed. That is a useful reminder to make stop criteria deliberate before execution.
For the checkout example, predeclare: affected-shard success rate, p99 end-to-end latency, queue age, retry ratio, duplicate-order count, and payment authorization mismatch. Add one guardrail for the people operating the system: stop on an active incident or on an operator's explicit request. A response-only metric is insufficient. If the gateway sheds load quickly, latency may improve while rejected orders rise.
| Signal | Example stop condition | What it protects |
|---|---|---|
| User outcome | Successful checkout below the agreed canary threshold for two minutes | Customers and conversion flow |
| Saturation | Queue age or worker utilization above a bounded threshold | Cascading backlog |
| Correctness | Any duplicate order or authorization-to-order mismatch | Financial and data integrity |
| Recovery | Queue fails to decline after the injected fault stops | A hidden recovery failure |
| Operations | Existing incident, paging escalation, or explicit operator stop | Team capacity and safety |
Threshold values are editorial examples, not defaults. Explain how the product owner and on-call team would set them from a service objective and known failure modes.
What counts as recovery evidence?
Recovery is not “the process is up.” A convincing answer traces the path from injected fault to customer outcome and back to steady state. Gather at least four kinds of evidence:
- Containment: Which target actually received the action? Did routing, partitions, and permissions keep the fault inside the intended boundary?
- Degraded behavior: Did the client receive the promised fallback, retryable error, or rejection? Was the response fresh enough and correctly labeled?
- System recovery: After the action ends, do queue age, in-flight requests, saturation, and tail latency return to their baseline trajectory without manual scaling that hides the problem?
- Business correctness: Do idempotency records, orders, payment authorizations, and downstream events reconcile? A retried request can return success twice while the ledger is wrong once.
Google SRE's overload discussion treats rejected work, retries, and request cost as system behavior rather than dashboard noise. A retry budget and idempotency key belong in an experiment when a client can retry a logical operation; otherwise a delay can become a retry storm that obscures the fault.
Make the result reviewable. Preserve the experiment version, target query, traffic shape, action start and stop time, alarms, dashboards, traces, and data reconciliation query. Compare predicted and observed results. If the claim failed, record whether the response should be code, capacity, configuration, or an explicitly accepted product trade-off. “We learned a lot” is not an operational conclusion.
Walk through a fault-injection answer
An interviewer says: “Design a test for a recommendation API that depends on a feature store. What happens if the store is slow?”
Ask whether recommendations are optional, whether stale results are acceptable, and whether feature reads happen synchronously. Suppose a response may use cached model output no more than ten minutes old, but the page must render. Your hypothesis becomes: a 500 ms delay on feature-store reads for one canary pod triggers a marked stale fallback; page rendering succeeds, fallback age is recorded, and retry fan-out stays within budget.
Stage the test. Validate the injector and dashboard in pre-production. In production, select one tagged canary pod and a short window, confirm the on-call owner, install stop conditions on page success, queue age, and fallback age, then introduce the delay. Compare canary and control traffic: response code, page completion, fallback rate, dependency latency, and saturation.
After removing the delay, prove recovery: fallback rate declines, queued work drains, no stuck circuit stays open, and sampled requests have an auditable reason for the fallback. If fallback age exceeded the promised limit even though pages were green, call the experiment a failed reliability claim. That calibration is often more impressive than asserting success.
What follow-ups expose shallow answers?
“How would you inject latency?” Choose the least invasive layer that faithfully represents the fault: a controlled proxy, test-only dependency mode, or platform action. Validate scope and reversibility in the target environment.
“What if the first alarm fires?” Stop the experiment, preserve timing and telemetry, assess whether guardrails worked, and return to steady state. Do not keep running to collect a more dramatic graph. A stopped run is evidence about the experiment's safety mechanism as well as the system.
“What if the test passes?” It may show one fallback path works for a particular delay and workload. It does not demonstrate resilience to every other failure mode. Propose the next smallest experiment based on remaining risk.
“Why not test only in staging?” Explain the trade-off rather than claiming production is always necessary. Staging usually gives safer repetition, while production can expose realistic routing, capacity, permissions, and workload interactions. Move only when the question cannot be answered safely in the earlier stage and the owners approve a bounded plan.
Practice with connected questions
| Question | Practice focus | Why it helps |
|---|---|---|
| Evaluate defensive testing and pair programming | Test choices | Explains why a test needs an observable claim |
| Identify and prevent code-breaking inputs | Failure modes | Narrows an input or dependency fault precisely |
| Improve robustness of graph cycle detection code | Defensive reasoning | Connects failures to concrete safeguards |
| Find a conflicting pair using a black-box run | Evidence collection | Practices testing an observed behavior before changing code |
| Write good tests and define integration tests | Test boundaries | Separates a small check from an end-to-end claim |
Frequently asked questions
Is fault injection the same as chaos engineering?
Fault injection is the act of introducing a defined failure. Chaos engineering is often used for a broader experimental practice: form a hypothesis, introduce controlled real-world variability, observe the result, and learn. The labels matter less than explicit safety, scope, and evidence.
Should a candidate recommend production testing?
Recommend a sequence: reproduce the behavior safely first, then explain the conditions under which a production experiment might be justified. Mention approval, pre-production validation, a small target, stop conditions, observability, and a named operator. Do not pretend that an interview grants permission to affect production.
Can an experiment use synthetic traffic?
Yes, if synthetic traffic represents the decision under test. It can isolate a fault and protect customers. State what it cannot show, such as real client retry behavior, authorization state, or shared capacity interactions. Complement it with controlled production evidence only when that remaining uncertainty matters and safeguards allow it.
What is the most common mistake?
Measuring only the injected error or host health. A system can restart cleanly while users face stale data, duplicate writes, delayed queues, or retry amplification. Tie the hypothesis and recovery evidence to a user outcome and data correctness boundary.
The best fault-injection answer is calm and specific: state what might fail, make the smallest experiment that can falsify the claim, stop before harm grows, and show how you would verify both service recovery and business correctness. That is the difference between a dramatic test and reliable engineering judgment.