Kafka Log Compaction Interview Questions: Tombstones, Retention, and State Rebuilds

A service rebuilds a customer cache from a compacted Kafka topic. Most values look correct, but a deleted customer reappears. The service reused an old cache, resumed after a long outage, and never observed the deletion marker. “The topic keeps the latest value” was not a complete recovery plan.

Kafka log compaction interview questions test whether you can connect keyed state, tombstone retention, background cleaning, and a consumer's starting state. Compaction supports state reconstruction under explicit assumptions. It is neither immediate duplicate removal nor a permanent record of every business event.

Evidence boundary: Official facts below use current Apache Kafka 4.3 documentation. The record trace, recovery calculations, and failure tests are original instructional examples. No candidate reports are used, and the practice questions do not establish employer-specific frequency. A local replay model checks the worked state transitions; no Kafka cleaner timing benchmark is claimed.

What does log compaction retain, and what can disappear?

Official fact: Compaction removes superseded records for the same key within a partition while preserving retained-record order and original offsets. A keyed record with a null payload is a tombstone; its marker can later be cleaned away too. Cleaning happens in the background. Kafka log-compaction design.

Original one-partition trace: Treat values as complete replacement snapshots and apply records in offset order:

Offset	Key	Value	State after applying the record
10	A	1	A=1
11	B	7	A=1, B=7
12	A	2	A=2, B=7
13	B	null	A=2; B deleted
14	C	9	A=2, C=9
15	A	3	A=3, C=9

A possible cleaned view retains offsets 13, 14, and 15. Replaying those records into an empty map also yields A=3 and C=9. Once B's tombstone is safely eligible for removal and cleaned, B may have no retained record at all.

Do not demand that this exact physical layout appear immediately. Check reconstructed final state; multiple physical versions can remain while cleaning is pending.

The snapshot assumption matters. If A's values mean increments of 1, 2, and 3, replaying only the final value produces 3 instead of the accumulated 6. A compacted changelog of complete totals has a different contract from an event stream of deltas. Decide which one you are publishing before choosing the cleanup policy.

Is a tombstone the same as the string null or an empty value?

Design inference from the null-payload rule: A deletion needs a non-null key and an actual null record value at the Kafka layer. The text "null", a JSON object such as {"deleted":true}, and an empty byte array are values with application-defined meanings. They do not become tombstones merely because a reader interprets them as deletion-like.

A serializer is therefore part of the correctness argument. Verify what bytes or null marker the producer emits and what the consumer receives. A test that checks only the object before serialization can miss a mismatch at the transport boundary.

An explicit deletion envelope can still be a valid design. For example, an application may retain deleted=true with a version to prevent stale updates from resurrecting a customer. That is a non-null latest value which the application interprets. Explain its storage and lifecycle consequences rather than calling it a Kafka tombstone.

The key bytes must also remain stable. Changing a customer's key encoding can create a new compaction identity. A tombstone written with the new key does not delete an older value stored under a different serialized key.

Does latest mean the newest business timestamp?

Original ordering counterexample: At offset 20, key A carries version 9. At offset 21, a delayed producer writes version 8. The later log occurrence is version 8 even though its business version is older. Compaction does not evaluate the version field and choose version 9 for you.

If the application's projection ignores older versions, a continuously running reader might correctly keep version 9. But after compaction removes the earlier record, a new reader may see only version 8. An application-level comparison during consumption cannot recover a newer value that is no longer present in its starting state or retained log.

Design inference: Enforce ordering before publishing the authoritative compacted state, or maintain a separate authoritative snapshot that resolves the conflict. Specify which producer owns updates for a key and how delayed events become ordered snapshots. Do not rely on record timestamps as an implicit compare-and-set.

Partition routing is equally important. If a key's updates move to a different partition after a partition-count or partitioner change, each partition can retain its own occurrence. A replay then lacks one simple per-key offset order across those partitions. Treat repartitioning a state topic as a migration with an explicit reconciliation plan.

Which settings control cleaning, retention, and tombstones?

Official fact: Topic settings separate cleanup policy, compaction eligibility, segment rolling, and deletion retention. compact,delete applies both mechanisms: old segments can expire, including segments containing the last remaining value for a quiet key. delete.retention.ms concerns tombstone retention, not a general TTL for live keys. Topic configuration.

Original configuration worksheet: Explain the question each setting answers before proposing numbers:

Setting	Operational question	Incorrect conclusion
cleanup.policy	Preserve compacted state, expire history, or both?	Any topic containing compact retains every live key forever
min.compaction.lag.ms	How long must new records remain uncompacted?	This is the maximum cleaner delay
max.compaction.lag.ms	When must records become eligible for compaction?	Every obsolete record disappears by this deadline
segment.ms / segment.bytes	When do active writes move into another segment?	A small topic is automatically cleaned immediately
delete.retention.ms	How long can readers rely on retained deletion markers?	Every live key expires after this interval

Official fact: The cleaner has broker-level enablement, thread, buffer, and I/O controls. Availability of cleaner resources affects when eligible work completes. Broker configuration.

Design inference: Diagnose “compaction is not working” by separating eligibility from execution. Inspect effective topic overrides, whether data is still in the active segment, and cleaner capacity. A configured eligibility limit is not a wall-clock guarantee that the physical files already contain one value per key.

For a state topic that must retain infrequently updated customers, adding time-based deletion changes the recovery promise. A customer whose last update is older than retention may disappear from a full replay despite never receiving a tombstone. That may suit a bounded cache, but it does not suit every authoritative directory.

Why can an old cache miss a deletion during recovery?

Official fact: Kafka's tombstone-retention setting bounds the scan period in which a reader starting from the beginning can rely on observing deletion markers. A sufficiently slow reader can miss markers while cleaning proceeds. Tombstone retention.

Original stale-cache scenario: A local store already contains B=7 from offset 11. It goes offline before offset 13's tombstone. Much later, both B's older value and its deletion marker have been removed from the retained log. Resuming the old local store does not encounter a record telling it to remove B. The stale B survives locally.

Contrast that with an empty rebuild after both records have disappeared. There is no old B to preserve, so B remains absent. However, a long-running rebuild can still race with tombstone removal after reading an old value. “Start empty” and “finish within the supported recovery horizon” address different failure modes.

Original sizing exercise: Suppose a tested restore must scan 180 million retained records and sustains 30,000 records per second while applying state. The idealized scan takes 6,000 seconds, or 100 minutes. Include startup, throttling, checkpoints, skewed partitions, and catch-up work before comparing the measured duration with your retention policy.

A count-based lag metric alone misses this risk. Ten million tiny records and ten million expensive state updates can take different times to restore. Measure elapsed rebuild time and the slowest partition's progress, then leave room for operational disruption.

How do you design a safe state rebuild and cutover?

Original recovery procedure: Choose either a verified snapshot with matching per-partition checkpoints or an empty state built from the retained beginning. A snapshot without its corresponding offsets cannot tell you where replay should resume.

Store state and replay progress with a crash-safe relationship. If an offset advances before its state update is durable, recovery can skip a needed change. If state advances before the offset, the record may replay; replacement updates and deletions should tolerate that replay. More complex effects need explicit idempotency.

Capture a target position for each partition so “caught up” has a measurable meaning. Apply ordered records, reconcile key counts and selected values against an authoritative reference, and switch readers only when the replacement state satisfies the chosen target. Keep the prior serving state available until the cutover is verified.

Official fact: Kafka's consumer API exposes seek operations and beginning/end positions. A compacted log can have gaps between offsets; a requested position need not correspond to a retained record. KafkaConsumer API.

Design inference: Check progress using consumer positions and the intended partition targets, not a requirement to observe every integer offset. Compaction never renumbers records to close gaps. At the same time, a position older than the available retention boundary needs a deliberate recovery decision rather than silent continuation from an arbitrary reset point.

For transactional source topics, choose the appropriate consumer isolation level. Compaction and transaction visibility solve different problems. A compacted topic does not by itself make aborted transaction records suitable for rebuilding committed business state.

What would you test before trusting the rebuild?

Original test plan: Replay the offset-10-through-15 fixture into an empty map, then replay the possible cleaned view. Both should produce A=3 and C=9. Next seed a stale map with B=7 and replay only A=3 and C=9; assert that B incorrectly survives. This proves why the starting-state assumption matters.

Test serialized tombstones, a late lower business version, repeated application of the same replacement record, and a key routed to another partition. For an integration test, observe cleaner progress rather than sleeping a fixed number of seconds and assuming cleanup has finished.

Official fact: Kafka publishes cleaner-related metrics, including measures of uncleanable partitions and compaction delay. Kafka monitoring. Pair those signals with restore duration and state reconciliation; reduced disk usage alone does not prove a correct recovery result.

Practice the retention and replay explanation

Practice question	Compaction follow-up
Explain Kafka partitions and delivery semantics	Explain partition-local order and why offsets retain gaps.
Reason About Kafka Consumers and Partitions	Define independent replay checkpoints for each partition.
Design a Kafka-Based File Watcher for Millions of Files	Decide whether a file update is a snapshot or a delta.
Design an Audit Logs Service	Separate immutable audit history from compacted current state.
Design a Distributed Logging System	Test retention, replay throughput, and stale local state.

A concise answer should name the key and partition contract, distinguish a tombstone from ordinary values, and trace the same records before and after cleaning. Then state the recovery starting point and retention horizon. That makes the guarantee testable instead of treating “latest value” as a complete specification.

Continue with the Kafka transactions guide to connect retained state with committed visibility and coordinated consumer progress.

Sources and Further Reading

• Kafka log-compaction design: retained state, tombstones, and offsets.
• Kafka topic configuration: cleanup policy and retention settings.
• Kafka broker configuration: cleaner resources and controls.
• KafkaConsumer Java API: replay positions and seeks.
• Kafka monitoring: cleaner health and delay signals.
• Managing Streams application topics: context for managed changelog topics.