Articles
Cassandra Tombstone Interview Questions: Deletes, Repair, and Read Amplification
Prepare for Cassandra tombstone interviews with delete and repair timelines, gc_grace_seconds limits, TTL trade-offs, and a practical slow-read diagnosis.
A Cassandra query returns ten rows but scans a large amount of deleted data. Someone proposes setting gc_grace_seconds to zero to make compaction remove tombstones sooner. Before discussing performance, explain what evidence that change could erase.
For Cassandra tombstone interview questions, the central idea is that a delete must outlive the old values it suppresses. A tombstone records that decision while replicas converge and local storage is reorganized. Elapsed time alone cannot tell you that every stale copy is harmless.
This guide follows a missed delete, separates repair from compaction, and turns a slow-read symptom into a defensible diagnosis.
Evidence boundary: Product behavior comes from Apache Cassandra’s stable documentation, labeled 5.0 when checked. The timelines, example timestamps, workload numbers, and recommendations are original interview exercises. They are not production measurements or verified employer interview reports. No independent recent candidate evidence establishes how frequently these exact questions appear. Confirm version-specific repair and compaction options for the deployment under discussion.
Why is a delete a write in Cassandra?
Official fact: Cassandra represents deletion with a timestamped tombstone that follows the write path. It suppresses older values rather than immediately erasing every physical copy. Expired TTL data also requires deletion handling before storage can be reclaimed. Apache Cassandra: Tombstones.
Imagine a replicated value that exists on three machines. If two machines merely erase their copies while the third is unavailable, the third later returns with a value and no visible evidence that it was deleted. Absence alone cannot tell the cluster whether a value is missing accidentally or intentionally removed.
The deletion marker supplies that missing information. In an interview, explain its purpose before naming a compaction setting. Otherwise “tombstones are garbage” can lead you toward removing the very record needed to preserve correctness.
Official fact: Cassandra resolves competing mutations using timestamps; CQL supports explicit mutation timestamps and TTLs, with different units and purposes. CQL data manipulation.
For the exercises below, 100, 200, and 300 are simplified mutation-order labels. They are not elapsed seconds or suggested production timestamps. A delete at 200 suppresses a value at 100. A legitimate later write at 300 is a separate event, not automatically a resurrected stale value.
How can deleted data return after a successful quorum write?
Original exercise: Use one data center, replication factor three, and replicas A, B, and C for the same partition. All initially store value V at timestamp 100. C becomes unavailable before the delete at 200 reaches it. A and B acknowledge the delete at QUORUM.
| Stage | Replica A | Replica B | Replica C | Meaning |
|---|---|---|---|---|
| Initial state | V at 100 | V at 100 | V at 100 | Replicas agree |
| C misses the delete | Delete at 200 | Delete at 200 | V at 100 | Quorum acknowledgment is possible without C |
| Safe convergence branch | Delete at 200 | Delete at 200 | Delete at 200 after repair | Old V is suppressed everywhere |
| Unsafe reclamation branch | Deletion evidence purged | Deletion evidence purged | V at 100 remains | A stale value can return without a newer marker opposing it |
The last two rows are alternative outcomes. They are not consecutive steps in one successful repair. That distinction matters: after the safe branch has propagated deletion evidence to C and removed the old value safely, the original missed-delete example no longer applies.
Inference from the documented deletion model: In the unsafe branch, later reconciliation can spread the old value because the surviving data no longer includes the delete that should defeat it. The problem is missing deletion history, not an old timestamp somehow becoming numerically newer.
Do not claim a quorum acknowledgment means every replica received the mutation. It tells you the configured response requirement was met. While deletion evidence remains available, a quorum read intersecting an acknowledging replica can reconcile against it; that reasoning does not justify discarding the evidence before all stale copies are handled.
If the interviewer changes the example to multiple data centers or LOCAL_QUORUM, redraw the acknowledgment set. Do not carry the single-data-center two-of-three explanation into a different replication topology.
Does gc_grace_seconds schedule immediate removal?
No. Treat it as a retention condition in the reclamation decision, not a timer that deletes a file at an exact second.
Official fact: A tombstone must be old enough and encounter a suitable compaction before it can be dropped. Older data in overlapping SSTables can prevent safe removal. When configured, only_purge_repaired_tombstones adds a repaired-data requirement. Tombstone removal conditions.
Original local-storage example: SSTable X contains delete 200. SSTable Y still contains V at 100 for the same partition. Removing the marker from X while leaving the old value available in Y could expose data that should remain deleted. Compaction needs enough information about overlapping data to establish that dropping the marker is safe.
Now distinguish that local decision from the remote replica problem. A node cannot infer that disconnected C learned the delete merely because its own SSTables no longer contain V. Replica convergence is an operational requirement; it is not automatically established by a local file rewrite.
Recommended answer to “Can we reduce grace?”: First establish maximum missed-write exposure, completed repair coverage, recovery procedures for long-unavailable nodes, and the table's mutation pattern. Then assess a change against those assumptions. If the team cannot complete convergence within the shorter window, lowering it creates a deletion-safety risk the performance chart does not show.
Conversely, a large grace period is not a substitute for repair. It gives the system more opportunity to converge, but opportunity is not evidence that convergence occurred.
How is repair different from compaction?
Official fact: Repair compares replicas over shared token ranges and streams differences. Hinted handoff is best effort and does not guarantee delivery of every missed mutation. The documented repair procedure must cover the relevant ranges; invoking repair on one node does not automatically establish complete cluster coverage. Apache Cassandra: Repair.
Compaction reorganizes SSTables on one node. Repair reconciles state between replicas. They interact, but they answer different questions: “Can I rewrite these local files safely?” and “Do the relevant replicas agree?”
Original operational follow-up: A team says, “Repair runs every Sunday.” Ask when it last completed for this table's ranges, whether any ranges failed, how long retries take, and whether the disconnected node participated or was safely replaced. A scheduled start is not a completed repair.
Suppose a grace window is ten days for the exercise, and a repair job starts weekly but sometimes needs four additional days to finish after failures. The nominal schedule does not establish a safe bound. Reason about the oldest unrepaired deletion and actual completion coverage, with room for recovery from failed jobs.
The 5.0 documentation distinguishes incremental and full repair. Incremental repair does not continually recheck data already marked repaired, so broader integrity problems may require a different repair plan. Avoid promising that one recurring command addresses every form of data divergence.
Your interview answer should include an owner for repair failures and an escalation condition. A dashboard showing job launches is weaker evidence than range-level completion records tied to the retention assumptions.
Why can LIMIT 10 still involve expensive reads?
LIMIT constrains the returned result, not the amount of obsolete data a storage engine must examine to produce it. Ten live rows can sit behind many expired cells, deletion markers, or overlapping versions.
Original workload: A partition stores a tenant's events indefinitely. The application repeatedly expires or deletes older events, then asks for the latest ten qualifying live rows across a wide clustering range. If large parts of that range are dead, returning ten rows may require substantial reconciliation and scanning.
The useful question is where the work occurs. Is one tenant responsible for oversized partitions? Does the query inspect a broad historical range? Are many SSTables involved? Is compaction falling behind? Is the apparent tombstone problem actually a different bottleneck, such as disk saturation?
Official fact: Cassandra exposes table metrics including TombstoneScannedHistogram, LiveScannedHistogram, tombstone warnings and failures, and partition-size measurements. These provide evidence about read work and data shape. Cassandra monitoring reference.
Do not turn one histogram into a universal safe threshold. Compare changes for the affected table and query population over the same interval, alongside latency and resource use. A cluster-wide average can hide one pathological partition.
| Observation in the exercise | Next evidence to collect | Safer direction to evaluate |
|---|---|---|
| Few returned rows, high tombstone scans | Exact query range and partition distribution | Narrow the range or redesign access paths |
| Many overlapping SSTables and rising latency | Compaction backlog and disk utilization | Resolve capacity or strategy mismatch |
| Old expired SSTables remain | Overlapping older data and expiry blockers | Understand why reclamation is blocked |
| A replica missed deletes for too long | Repair coverage and node-recovery history | Preserve deletion correctness before aggressive cleanup |
Raising warning or failure thresholds does not reduce the scan. It may permit more work to continue and conceal the symptom. Explain what evidence would justify a temporary threshold change, what resource budget limits it, and what actual data-model or operational fix follows.
When do TTL and time-window compaction help?
Official fact: Time Window Compaction Strategy groups SSTables into time windows and can make expiry efficient for suitable time-series workloads. Mixing old and new data, or different expiry behavior within files, can make whole-file expiration harder. TWCS documentation.
Original redesign: Replace an unbounded tenant partition with tenant-and-day buckets when the application's queries are naturally time-bounded. Apply a consistent retention policy to that event stream, and query only the buckets needed for the requested interval. Evaluate TWCS against the resulting write and expiry pattern rather than enabling it solely because the table has TTLs.
This trades one large scan for bounded partitions and potentially multiple reads. A request for the latest ten events may need to inspect today's bucket and then earlier buckets until it has enough results. Define a maximum lookback and pagination behavior; bucketing is not useful if the application immediately scans every bucket forever.
Late arrivals deserve their own test. An old event written today can interact differently with time windows and expiration than an event arriving on time. Decide whether retention follows event time or ingestion time, and make the application calculate any required remaining TTL consistently.
TTL expiration makes data logically unavailable; it does not promise instant physical disk recovery. Measure the delay between expiry and reclamation under a representative workload. Include late writes, updates, and repair activity instead of benchmarking only a clean append-only stream.
How should you answer the final troubleshooting prompt?
If asked, “Reads are slow and there are many tombstones—what do you do?”, begin with the affected query and correctness constraints. Identify the table, partition shape, scan range, latency change, and replica health. Then distinguish excess read work from blocked reclamation or insufficient compaction capacity.
State one proposed change and one way it could fail. Narrowing a query might omit required historical results. Reducing grace might outrun repair. Changing compaction may not fix an unbounded partition. The stronger answer makes those trade-offs visible and names the measurement that would validate the improvement.
Sources and Further Reading
- Apache Cassandra: Tombstones
- Apache Cassandra: Repair
- Apache Cassandra: Time Window Compaction Strategy
- Apache Cassandra: Monitoring
- Apache Cassandra: CQL data manipulation
Research checked September 21, 2026. Product behavior and supported settings can vary by version and deployment.