Articles
Ceph Interview Questions: Placement Groups, CRUSH, and Degraded Cluster Recovery
Prepare for Ceph interviews with PG and CRUSH mapping, size versus min_size, host-failure exercises, and evidence that separates degraded I/O from recovery.
A Ceph cluster can have plenty of live OSDs and still fail to restore a missing replica. The free space may sit on the wrong hosts, a placement group may lack required write history and remain inactive, or backfill may be blocked by a destination's capacity threshold. A healthy-OSD count does not identify each PG's copies, history, or legal recovery targets.
For Ceph interview questions, trace one placement group from its pool configuration to its eligible failure domains, then separate serving I/O from restoring redundancy. This guide uses Ceph's Squid documentation as an explicit release reference and original recovery exercises. Check documentation for the deployed release before applying commands or defaults. Candidate reports do not establish a particular employer's syllabus or incident procedure.
Start with Design a replicated cloud storage service, then explain which replicas and placement constraints support its durability claim.
How do objects map through PGs and CRUSH?
Official architecture: Ceph's placement-group documentation explains that placement groups collect objects within a pool and provide a manageable unit for placement and recovery. An object's pool and identity participate in its PG mapping; CRUSH determines storage placement using the cluster map and placement rule. A PG is not a physical disk or a one-object container.
Official placement rules: the CRUSH map documentation describes a hierarchy of devices and buckets, along with rules that choose devices across specified failure domains. A rule can require replicas on different hosts. Device class and the selected hierarchy also constrain eligible targets.
Check two kinds of capacity: aggregate free bytes and free bytes on devices the rule can choose. A new OSD on an already-used host can add storage without adding a distinct host failure domain. Conversely, an eligible host with insufficient usable headroom may satisfy topology but still be unable to accept the intended recovery work.
Do not describe CRUSH as a central lookup table containing one row for every object. Also avoid claiming that its placement guarantees equal byte usage. Objects and pools differ in size and workload; counting PGs per OSD is not the same measurement as comparing bytes or I/O demand.
What do size and min_size actually mean?
Official pool contract: Ceph's pool documentation defines replicated-pool size as the total desired number of copies, including the primary copy. size = 3 means three total copies, not an original plus three more. min_size sets the minimum replica count required for I/O under the pool's configuration.
For the original exercises, set size 3, min_size 2, and require three distinct hosts for placement. Assume the surviving replicas contain the authoritative data, peering can complete, and no additional fault or policy blocks the operation. These assumptions are part of the exercise; the numbers alone do not establish them in a live cluster.
With two valid copies, a PG can potentially remain available while below its desired redundancy. With only one valid copy, it does not meet this pool's I/O minimum. Neither statement is a universal rule for every Ceph pool: an erasure-coded pool has shard and recovery requirements that cannot be replaced with this replicated-copy arithmetic.
Separate durability from availability. Data may still exist on an OSD that is currently unreachable. A PG may be unable to serve requests while recovery remains possible. Conversely, an available degraded PG is operating with less protection than intended. “The cluster responds” does not mean every PG is available or fully replicated.
Original host-failure case: can the third copy be restored?
Use four hosts, each with two OSDs:
Host A: osd.0, osd.1
Host B: osd.2, osd.3
Host C: osd.4, osd.5
Host D: osd.6, osd.7
Our example PG 8.a initially has its three copies on osd.0, osd.2, and osd.4, spanning A, B, and C. This mapping is supplied for reasoning; it is not the result of a CRUSH execution performed for this guide.
Host A fails. Two authoritative copies remain on B and C. Under the stated peering and health assumptions, the PG can meet min_size 2 while its third copy is missing. If D has eligible devices and sufficient headroom, it offers a distinct host on which a replacement copy can be placed.
Now remove D from the original cluster design. After A fails, only B and C remain, even though each contains two OSDs and substantial free space. A host-level rule cannot choose three distinct live hosts from two. Adding another OSD to B does not resolve that constraint. You need an eligible additional failure domain or a deliberately reconsidered protection policy; spare bytes on B are not a third host.
| Original case after A fails | Live eligible hosts | What the exercise establishes |
|---|---|---|
| A/B/C/D design, D has headroom | B, C, D | A legal third-host target exists |
| A/B/C design | B, C | Two copies may serve, but three-host placement cannot be restored |
| A/B/C/D, D excluded by the pool rule | B, C | D's raw capacity does not satisfy this placement rule |
| A/B/C/D, D cannot accept backfill | B, C, D | Topology is sufficient; recovery capacity is not established |
After identifying a legal target, verify that backfill actually completes. Monitor the actual PG and recovery progress. If B fails before the third authoritative copy is established, only C remains in this simplified case and min_size 2 is no longer met. Do not count a target selected in a map as a finished copy.
Which PG states imply different problems?
Official state meanings: Ceph's PG-state reference distinguishes active, where requests can be processed, from clean, where the intended object replication is complete. degraded indicates objects missing their intended copies; undersized indicates fewer copies than the configured replication level. peering concerns establishing the PG's shared state and history.
Read combinations and affected PG counts. An active degraded PG and an inactive PG present different immediate customer impact. A state involving recovery or backfill describes work, while waiting or fullness-related states can explain why that work is not progressing. Do not reduce every non-clean state to “data loss.”
Also distinguish misplaced data from missing data. A replica can still exist on an OSD outside the current intended placement while movement is pending. That is not the same condition as having fewer valid copies. During remapping, current serving placement and desired placement can differ; inspect the relevant sets and history before deciding what has been lost.
In the original case, “active and degraded” is a possible outcome under the assumptions, not a fabricated ceph -s transcript. The actual state can differ while peering, mapping changes, or additional health conditions are unresolved. An interview answer should propose how to observe the state that selects its next action.
How would you diagnose a degraded cluster before changing it?
Begin with scope: which application, pool, and PGs are affected, and are clients failing requests or experiencing latency? Capture the time of the observation and recent topology changes. Then collect read-only evidence, using actual pool and PG identifiers:
ceph -s
ceph health detail
ceph osd tree
ceph osd pool get data size
ceph osd pool get data min_size
ceph osd pool get data crush_rule
ceph pg 8.a query
The example identifiers data and 8.a are placeholders from the teaching case. These commands are inspection examples; this guide executed no Ceph commands against a cluster. Pool getters identify the configured contract, the tree reveals failure domains, and the PG query helps investigate its current recovery or peering condition.
Use the rule identifier to inspect the associated CRUSH rule, including its root, device class, and failure-domain choice. A host's mere presence in the cluster tree does not prove the pool can choose its devices. Check utilization on eligible targets, not just a cluster-wide free-space total.
Then compare observations over time. Is the degraded-object count falling? Are the same PGs still waiting? Are OSDs flapping, increasing uncertainty about the authoritative history? Is foreground latency deteriorating while recovery runs? A later snapshot adds evidence about movement; repeated identical screenshots do not establish progress.
This diagnosis sequence is engineering guidance inferred from the documented configuration and states. It does not prescribe a universal corrective command. The next change depends on whether the evidence points to failed hardware, missing placement domains, fullness, communication failures, or unavailable history.
Why can backfill stall when the cluster has free space?
Official health behavior: Ceph's health-check reference explains that an OSD at the backfillfull threshold, or projected to exceed it after mapped backfills, can prevent rebalancing to that OSD. The relevant constraint is destination utilization, not merely total cluster free capacity.
Extend the four-host exercise: D is the only eligible new host for this PG, but its eligible OSDs cannot accept the planned backfill. Free capacity on B and C does not substitute for D under the three-host rule. Adding a device outside the rule's selected class also does not create a usable target for this pool.
Inspect the specific health warning and target utilization before changing thresholds or flags. A paused backfill may be protecting a destination that cannot accept more data. Making a warning disappear by relaxing a protection does not demonstrate that recovery can complete safely. Describe the proposed effect and the evidence needed to validate it.
For performance, distinguish waiting for a legal destination from moving data slowly. More recovery concurrency cannot supply the missing third host required by this pool's rule. Conversely, a valid target does not establish adequate bandwidth or healthy source reads. Match the remedy to the limiting condition rather than applying every tuning knob.
Should you lower min_size to restore service?
Changing min_size changes the pool's availability and protection trade-off; it does not create another copy or recover unavailable history. In our example, lowering the minimum from two to one would permit a different I/O policy with only one copy, but it would not restore the missing replicas.
The official pool documentation warns that min_size 1 or size 2 in production introduces data-loss risk and limits such choices to exceptional circumstances. An interview answer should therefore identify the exact unavailable PGs, remaining authoritative information, recovery options, and business decision before presenting a policy reduction as a remedy.
Avoid confusing an emergency access decision with a completed recovery. State what can still fail, how writes affect the remaining protection, and what observation would establish the intended redundancy afterward. If the PG lacks necessary history, a replica-count setting alone cannot manufacture that evidence.
A better response to an underspecified prompt is conditional: “I would first inspect this PG's state and history. If it has two authoritative copies and can peer, it may serve under min_size 2. Restoring size 3 also requires a legal target and successful data movement.” That answer states both the available path and its limits.
How should you discuss PG counts and autoscaling?
PG counts balance placement granularity against management and recovery overhead. They are not a direct repair for missing host domains or an incomplete PG. Before proposing a count change, inspect the pool's autoscaling mode, data distribution, eligible OSDs, and operational context.
Ceph's release documentation describes autoscaling and manual settings. Do not memorize one PG count as suitable for every pool, or assume that increasing it immediately fixes byte imbalance. A placement change can itself require movement. Explain the measurement that motivates it and the cost of carrying it out while the cluster is already degraded.
Keep this answer distinct from object-level application sharding. A client-facing key space, a Ceph pool, and a PG are different abstractions. Connecting them correctly demonstrates more understanding than listing configuration names.