Articles
Elixir OTP Interview Questions: Supervisors, GenServers, and Failure Recovery
Explain Elixir OTP supervisors, GenServer calls and casts, restart strategies, state loss, and durable failure recovery with a practical scenario.
An Elixir OTP interview question often asks what happens after a GenServer crashes. “The supervisor restarts it” is only the first sentence. A complete answer names the child specification, restart strategy, what happens to callers and in-memory state, and how the application repairs any unfinished external work. Restarting a process does not automatically restore its last state or make a payment, email, or message happen exactly once.
Consider a GenServer that tracks a delivery batch in memory, writes completed items to a database, and sends notifications through an external provider. A malformed message crashes its callback after the provider accepted a notification but before the database records completion. The supervisor may start a fresh process, yet the provider side effect is uncertain. In an interview, draw the process tree and the durable operation timeline separately. Practice the recovery-design side with Design a Fault-Tolerant Distributed Job Scheduler.
Evidence boundary: Official facts about Supervisor, GenServer, DynamicSupervisor, and processes come from current Elixir documentation checked September 2026. Candidate reports are not used to predict a company's exact OTP exercise. Inference includes the delivery example, suggested persistence choices, and test plan; those depend on the application's contract.
What does a GenServer own?
A GenServer is a process with callbacks for initialization and messages. init/1 establishes its initial state; handle_call/3 handles synchronous requests, handle_cast/2 handles asynchronous casts, and handle_info/2 handles other messages such as ordinary send messages and monitor notifications. Its state is an Elixir term held by that process. Other processes interact by messages rather than mutating that state directly.
The official GenServer guide emphasizes that call waits for a reply and supplies useful backpressure. cast sends asynchronously when the caller does not require a response; it does not guarantee that the server received or processed the message. This matters in failure recovery. If a caller needs to know whether a command was accepted, using a cast and assuming success is a weak contract. If a call times out, however, the caller also must not infer that the server did nothing. The server might have completed an external side effect before its reply was lost or delayed.
| API/message | Caller observes | Server callback | Recovery question |
|---|---|---|---|
GenServer.call | Reply, timeout, or exit | handle_call/3 | Did work finish before the reply failed? |
GenServer.cast | Send returns without a server reply | handle_cast/2 | Was the command processed at all? |
Ordinary send | Message sent without a typed reply | handle_info/2 | Does the process handle unexpected messages safely? |
Do not put every calculation in a GenServer just because OTP exists. Elixir's Supervisor docs themselves use a counter for teaching but note that a simple counter often should be passed through ordinary functions instead. A GenServer is useful when one process must coordinate state, sequence messages, own a resource, or serve a lifecycle. A pure transformation generally stays simpler as a function.
What exactly does a supervisor restart?
A supervisor owns child specifications and watches child processes. A child specification defines how a process starts and includes restart and shutdown behavior. Elixir documents three restart values: :permanent restarts after any termination, :transient restarts only after abnormal termination, and :temporary never restarts. The default is :permanent. State which one applies before predicting a restart.
The strategy determines which other children are affected when one child needs restarting. Suppose children are started in order A, B, C, and B crashes:
| Strategy | Processes restarted after B's qualifying failure | When it can fit |
|---|---|---|
:one_for_one | B | Siblings are independent |
:rest_for_one | B and children started after it, including C | Later children depend on B |
:one_for_all | A, B, and C | The group must restart coherently |
That table is not a reason to choose a broad restart by default. If C depends on a resource owned by B, :rest_for_one may be sensible. If A, B, and C are unrelated users' sessions, restarting all of them after one bad input would widen the blast radius. Draw the dependency direction and choose the smallest restart scope that restores a valid state. A DynamicSupervisor is appropriate when children are added dynamically; it does not change the need for explicit state and side-effect recovery.
Supervisors also limit repeated restarts. The current Supervisor docs describe restart intensity controls such as :max_restarts and :max_seconds; when a child repeatedly fails beyond the allowed intensity, the supervisor itself terminates, allowing a parent supervisor to handle the broader failure. “Let it crash” is therefore a deliberate strategy for isolating unexpected failures inside a hierarchy, not a promise of infinite silent retries. A poison message or broken dependency can cause a restart loop and escalate.
What state survives a GenServer crash?
The process's in-memory state ends with the process. A restarted GenServer runs its start path and init/1 again, producing a new state from its arguments or whatever durable source it reads. The Supervisor example in Elixir's docs demonstrates a counter returning to its initial value after a crash. If your design needs the last committed delivery-batch position, store it durably and load it during startup or reconstruct it from events. Do not describe process restart as state rollback to the last good in-memory value; it is a new process.
For the delivery example, define three layers. Ephemeral state may hold a cache or currently processed item. Durable state records which batch items are committed. External state is what the notification provider accepted. On restart, rebuilding ephemeral state from durable records is possible, but an ambiguous provider response still needs reconciliation. A stable operation key and a provider idempotency feature can help if supported. If not, business policy must define whether a duplicate notification is acceptable, detectable, or compensatable.
Imagine this exact sequence: B processes item 42, provider accepts it, then B crashes before database commit. The supervisor starts B again. If B loads a database row that still says item 42 is pending and sends it again, the process tree has recovered but the business outcome may be duplicated. A good interview answer names that gap and proposes a test around it. It does not call the supervisor a transaction manager.
What happens to the caller when the server fails?
A synchronous caller can receive an exit or timeout. The documented GenServer.call contract says it waits for a reply unless it times out or nodes disconnect. If the target process dies during the call, the caller must handle that failure according to its own boundary. It should not blindly retry a non-idempotent command without checking whether the previous attempt affected durable or external state.
For asynchronous work, a cast's lack of reply is even more important. A caller that writes “sent” to its own database immediately after cast has only observed that it sent a message, not that the GenServer processed it. If the operation needs durable acceptance, place it in a durable queue or use an acknowledgement protocol. If it is a best-effort telemetry update, losing it may be acceptable. State the required guarantee first, then pick the API.
Monitoring a process gives :DOWN information when it terminates. Linking changes how exit signals propagate. Those are BEAM process tools, but they do not themselves make an arbitrary message durable. In an OTP interview, separate observing a failure from replaying work correctly. The Erlang-focused neighboring article can go deeper on raw monitor and mailbox semantics; here the primary design question is how a supervised GenServer resumes application responsibility.
How would you test the supervision design?
Start the real small supervision tree in a test, not an isolated callback module. Capture the initial child PID, trigger a deliberate failure, and verify that a new PID appears when the child specification requires a restart. Assert which sibling PIDs changed under the chosen strategy. For A/B/C, a :one_for_one test should show A and C unchanged after B's failure; a :rest_for_one test should show C restarted too. These assertions prove the strategy actually matches the dependency story.
Then test state recovery. Persist one completed item, crash the GenServer, and verify that the restarted process reconstructs the intended progress. Inject a crash after the external provider accepts but before the local commit. Check the final provider-call count and durable operation record. If the contract requires one notification, a test that only proves a new PID exists is insufficient.
Test a poison input separately. If the same bad message recurs and every restarted child immediately crashes, observe the restart-intensity behavior and alerting. A system that continually restarts without making progress is unavailable even if each restart is technically successful. Instrument child restart counts, error reasons, queue age, and a business completion measure. Recovery evidence is the restored service and correct outcome, not a green supervisor status alone.
How do you choose boundaries in a real tree?
Group children by failure dependency and ownership. A database connection pool shared by several workers may be an ancestor-level resource. A single customer's session is better isolated from other customers' sessions. A DynamicSupervisor can start many similar child processes on demand, but high cardinality still needs resource limits and cleanup. If each child holds expensive state, blindly spawning one per request is not automatically a good design.
Shutdown also matters. Child specs include a shutdown timeout; the supervisor sends a shutdown signal and waits before forcing termination. A worker that must finish an external operation needs a bounded, observable shutdown path and durable resume plan. Setting an infinite shutdown timeout on an ordinary worker can prevent the application from terminating; the docs caution against it. Pair graceful termination with crash-safe idempotency rather than trusting only one path.
What are the most common interview mistakes?
One mistake is equating :one_for_one with “the whole system is resilient.” It limits the restart to one child, but that child's state and side effects still need a recovery model. Another is confusing the :restart value with the supervisor strategy: :temporary can prevent restart even under a broad strategy. A third is treating cast as reliable completion acknowledgement. A fourth is putting all business state in a GenServer when the process can crash and the state must survive.
The strongest answer narrates a specific failure boundary. Name the message, the callback, the child spec, which PIDs change, what the caller observes, what state is reconstructed, and how an ambiguous external effect is resolved. If you can describe those in order, “let it crash” becomes an engineering policy rather than a slogan.