Articles
HDFS Interview Questions: Write Pipelines, Under-Replicated Blocks, and NameNode Failover
Practice HDFS interview questions with write-pipeline traces, under-replicated block evidence, safe diagnostics, and a separate NameNode failover timeline.
Strong HDFS interview questions ask you to follow a failure across several boundaries: the client’s write pipeline, the available block replicas, and the NameNode service that manages the namespace. Keep those boundaries separate. A block can retain readable copies while a client cannot obtain metadata, and a recovered NameNode does not automatically restore the intended replication factor.
The most useful answer names what you know, what remains uncertain, and which observation would settle it. “HDFS has three replicas, so it is safe” skips the actual state of the system.
Start with Design a Distributed File System for broader design practice. Here, we work one block through a DataNode failure and a separate NameNode failover, using independent evidence ledgers.
Establish the version and storage assumptions
Official baseline: The references in this article are pinned to Apache Hadoop 3.4.2. We use ordinary replicated HDFS blocks, not erasure-coded files. Hadoop’s architecture documentation assigns namespace and block-location decisions to the NameNode and block storage and transfer to DataNodes.
Original practice assumptions: A file contains block B7, its target replication factor is three, and its initial replicas are on D1, D2, and D3. The client has working credentials and network access. Later, a separate failure affects the Active NameNode. Host names, packet numbers, and the timeline below are invented for rehearsal.
These assumptions matter. An authentication failure can resemble unavailable storage to an application. An erasure-coded block group requires a different reconstruction explanation. A single-NameNode deployment cannot inherit the failover guarantees of a configured HA deployment.
Candidate reports are not used to establish these technical behaviors or their interview frequency. The questions and diagnostic sequence are our preparation guidance, grounded in the linked implementation and documentation.
Trace metadata separately from the block write
For a three-DataNode pipeline, the client sends block data to D1, which forwards it to D2, which forwards it to D3. The NameNode selects locations and handles metadata; it is not a proxy through which the file’s bytes travel. This separation follows the HDFS architecture.
Draw two kinds of arrows when explaining this. One connects the client and NameNode for namespace operations and location information. The other connects the client and the DataNode chain for packet data. Avoid a diagram that places the NameNode between the client and D1 on the data path.
For our case, ask three questions about packet 42: Was it transmitted? Was it acknowledged through the current pipeline? What does the application’s flush or synchronization operation promise? These questions describe different milestones.
An acknowledgment is also scoped to the pipeline that produced it. If the pipeline later loses a member, the target replication factor of three does not prove that every subsequent packet has three healthy copies. Read the recovery state before making that claim.
What changes when D2 fails mid-write?
Implementation evidence: Hadoop’s tagged DataStreamer source tracks sent packets awaiting acknowledgment. Its recovery description removes the failed DataNode, rebuilds the pipeline, and resends outstanding packets. That makes an unacknowledged packet uncertain from the client’s perspective, rather than automatically lost.
| Original case event | Defensible conclusion |
|---|---|
| Packet 42 received a successful pipeline acknowledgment before failure. | Treat 42 as acknowledged through that pipeline; do not extend that fact to 43. |
| D2 fails while43 is outstanding. | The client cannot infer 43’s final status from “sent” alone. |
| Recovery excludes D2 and reconnects surviving participants. | Inspect recovery success and replacement policy before promising continued writes. |
| Outstanding data is retried and 43 is acknowledged. | Record the successful recovered write path separately from the long-term replication target. |
| A healthy replacement later receives B7. | Confirm the replacement replica through current block evidence. |
Configuration fact: The 3.4.2 defaults expose DataNode replacement policy and failure behavior. Replacement is not an unconditional “a new node always joins immediately” rule.
In the rehearsal, explicitly assume that the configured policy and available nodes permit the selected recovery path. If the interviewer removes that assumption, explain the branch: replacement may succeed, a reduced pipeline may be allowed, or the write may fail. “Replication factor three” alone cannot choose among those branches.
Do not describe retransmission as an application-level duplicate append. Packet recovery belongs to the stream protocol. If the application independently retries an entire failed file operation, discuss its path naming, completion checks, and retry semantics as a separate problem.
Does acknowledgment mean power-loss durability?
Avoid turning a packet acknowledgment into a universal guarantee about physical disk persistence. The Syncable API distinguishes hflush, which makes buffered data visible to new readers, from hsync, which synchronizes toward the disk device and notes possible device caching.
Our follow-up exercise: Suppose the client saw packet 42’s acknowledgment and then an entire rack lost power. What additional facts would you need? Identify the applicable synchronization operation, the replicas’ failure domains, and the actual storage persistence guarantees. A network acknowledgment alone does not answer every part of that question.
Likewise, successful stream progress and successful file completion are different observations. Ask whether the file remains open, whether close completed, and which bytes a new reader can observe under the relevant operation. Do not casually treat a partly written file as an ordinary completed block in your diagnostic ledger.
This is a useful place to state an uncertainty precisely: “I can establish the client’s acknowledgment state from the stream evidence; I need the synchronization and storage assumptions before asserting survival of power loss.”
Under-replicated is a risk state, not a synonym for lost
Original completed-block ledger: After the write is successfully completed, suppose B7 has two healthy, reachable replicas on D1 and D3 against a target of three. The file remains readable under our connectivity assumptions, but its redundancy is below the target. If only corrupt or unavailable copies remain, the conclusion changes.
| Observed B7 state | Next question |
|---|---|
| Two healthy copies, target three | Why has the remaining copy not been rebuilt? |
| Three copies on an unsuitable placement | Does the placement meet the configured failure-domain policy? |
| A host has a heartbeat, but B7 is absent from its report | Does this host actually provide the required block replica? |
| No healthy reachable copy | Is there another usable replica or an independently recoverable copy? |
The HDFS Users Guide distinguishes diagnostic reporting from repair and explains that safe mode restricts modifications. Our diagnostic guidance: Check the healthy source, eligible destination capacity, placement constraints, and repair progress before proposing a recovery action.
For B7, adding D4 to the cluster is only an enabling event. It does not prove that D4 holds B7. A scheduled transfer is not a verified completed replica either. Update the ledger when the relevant observation changes, not when an administrative action sounds promising.
Ask whether D1 and D3 share a failure domain. Two copies can provide less independent protection than the count suggests. Conversely, a temporarily low replica count is not enough evidence to declare a lost file. Explain the observed risk without escalating the language beyond the evidence.
Use a read-only diagnostic sequence
The commands guide documents the following inspection commands. Substitute the real path and configured NameNode IDs; these examples were not executed against a Hadoop cluster.
hdfs fsck /warehouse/events -files -blocks -locations
hdfs dfsadmin -report
hdfs dfsadmin -safemode get
hdfs haadmin -getServiceState nn1
hdfs haadmin -getServiceState nn2
First narrow the affected file or directory and inspect block-level evidence. Then compare DataNode availability and capacity. Finally, check namespace service state when metadata operations fail or reports cannot be obtained.
The sequence is a rehearsal approach, not a requirement to run every command in that order. If the client cannot reach the namespace service, record that limitation before treating a missing diagnostic result as proof that a block is gone.
Avoid adding mutating fsck options to an inspection command by habit. Reporting a problem, triggering replication work, moving files, and deleting files are distinct operations. Name which action you propose and the evidence that justifies it.
For our B7 case, a useful answer ends with a testable next observation: “I will check whether a valid source and eligible target exist, then verify that the new replica is reported and healthy.” “I will restart everything” provides no comparable diagnosis.
Keep the observation time beside each entry. A replica count captured before D2 failed and a service-state report captured after failover describe different moments. Aligning them prevents an apparently contradictory answer that merely combines stale evidence with current evidence. Say which ledger entry you are refreshing and why.
What QJM protects during NameNode failover
Official HA facts: In the QJM HA configuration, namespace edits are recorded to a majority of JournalNodes. A Standby catches up before promotion. QJM restricts journal writing to one NameNode; fencing remains relevant because an old Active may still serve stale reads. Automatic failover adds ZooKeeper and ZKFC, and clients require HA-aware configuration.
Original namespace exercise: We have J1, J2, and J3. A majority requires two participants. With J1 unavailable but J2/J3 reachable, the numerical quorum remains possible. With only J3 reachable, it does not. This is quorum arithmetic; it is not proof that all health, networking, or promotion conditions are satisfied.
Keep another ledger for the candidate Active: observed health, edit catch-up, writer ownership, and client routing. If a process is running but has not completed the necessary transition, “the second NameNode is up” is insufficient evidence of successful failover.
Now return to B7. Promoting the Standby changes namespace availability. It does not itself create the third data replica. Conversely, copying B7 to D4 does not repair a namespace quorum or a broken client failover configuration. The two recoveries can succeed at different times.
Why a SecondaryNameNode is not the HA answer
The Users Guide describes the SecondaryNameNode’s checkpoint work: merging namespace image and edits. The HA guide describes Standby checkpointing within an HA deployment.
Our interview follow-up: If someone says “the SecondaryNameNode automatically replaces the failed NameNode,” ask what configured promotion mechanism, edit synchronization, and client routing they mean. A component’s reassuring name cannot establish those capabilities.
In the original incident, the immediate goal is to restore a valid namespace service and then verify client access. A checkpoint may be relevant to a different recovery scenario, but that does not make it an interchangeable automatic Standby. Explain the particular deployment you have rather than combining every HDFS recovery component into one story.
Five questions to connect failure evidence to design
The database failover question is especially useful for comparison: identify which invariants transfer and which implementation details do not.
| Complete practice question | HDFS follow-up to explain |
|---|---|
| Design a Distributed File System | Separate metadata availability from block-data transfer and durability. |
| Design a Distributed File System, Scoping the Requirements Yourself | State replication, write-completion, and failure-domain assumptions. |
| Design a replicated cloud storage service | Explain why target replica count differs from verified healthy copies. |
| High Availability and Failover for a Primary-Replica Database | Compare promotion and single-writer safety without equating database logs with HDFS block replicas. |
| Optimize MapReduce performance | Distinguish storage failures from locality and job-execution problems. |
Rehearse Design a Distributed File System using B7 and both ledgers. After every proposed action, name the observation that would prove the intended recovery actually happened.