Articles
Kafka Interview Questions for Data Engineers: Partitions, Consumer Groups, and Delivery Semantics
Practice Kafka interview questions for data engineers: partition ownership, safe offset commits, delivery semantics, lag diagnosis, and database replay safety.
Adding Kafka consumers can leave throughput unchanged. A database can receive duplicate updates even when the producer uses idempotence. And a consumer can report progress while an earlier record remains unfinished. These are useful starting points for Kafka interview questions for data engineers because they expose the difference between reading events and safely applying them.
The central answer: explain who owns each partition, which effects are durable, and where processing resumes after failure. Attach delivery guarantees to that complete path.
This guide uses official Apache Kafka documentation, checked on September 30, 2026. The workloads and failure exercises are original preparation examples, not candidate reports or predictions of an employer's interview. We focus on ordinary subscribed consumer groups; share groups and manual partition assignment have different coordination rules.
Give an answer aloud, then change the number of partitions and the crash point.
How do partitions and consumer groups divide the work?
Official fact: ordinary consumer groups distribute subscribed partitions among their members. Within a group, a partition has one consumer owner at a time; separate groups maintain independent consumption progress. Kafka retains topic records according to retention configuration rather than deleting each record when one consumer reads it. Apache Kafka introduction
Consider one topic with three partitions and five instances in one group. Assuming healthy members and a balanced assignment across that single topic, three instances can own one partition each and two remain idle. The exact identities depend on assignment behavior. A second group can independently consume the topic for another purpose.
A strong follow-up answer separates two changes. Adding instances to the same group creates potential processing capacity but cannot make five instances independently own three partitions. Giving every instance a different group ID makes each group eligible to read the whole topic; it does not divide one workload five ways.
Also distinguish partition count from replication factor. Replicas support durability and availability; they are not additional partition assignments that ordinary consumers use to split the same records.
What ordering can you promise?
Official fact: Kafka provides append order within a partition. Its introduction describes how records sharing a key can be directed to the same partition. That does not establish a global order across the topic. Partitioned logs and keys
Design inference: choose a key that matches the business operation requiring serialization. An inventory stream might use a warehouse-and-item key if updates for that pair must be applied in sequence. A warehouse-only key could concentrate too much traffic in one partition.
State the assumptions behind your promise: stable partition routing, a suitable partitioner, and processing that preserves the required order. Increasing partition count may change key placement. Launching asynchronous writes from records read in order may also make database effects complete out of order.
Append order is not event-time order. A delayed update can arrive after a newer event. If the sink maintains current state, define how a source version or sequence number prevents an old update from overwriting a newer one. Committing offsets correctly does not solve that business conflict by itself.
Why might more consumers fail to reduce lag?
Use this synthetic capacity exercise. Partition P0 receives 800 records per second; P1 and P2 each receive 100. Each partition's processing path can sustain 500 records per second under the stated workload.
P0's backlog grows by 300 records per second. The other two paths have spare capacity, but ordinary group assignment cannot lend their ownership to a second consumer for P0. Adding five idle consumers changes neither that partition's arrival rate nor its processing rate.
If arrivals stop and P0 has 18,000 outstanding records, processing at 500 per second takes 36 seconds, assuming constant service rate and no other bottleneck. If arrivals continue at 800 per second, the consumer cannot catch up at that capacity. Quoting a catch-up time without subtracting continuing arrivals is a reasoning error.
Before scaling, inspect per-partition lag, key distribution, payload sizes, processing latency and sink contention. A hot key calls for a different discussion than uniformly slow database writes. Splitting a business key may improve parallelism while sacrificing the ordering the application needs. Explain that trade-off before recommending repartitioning.
Which offset is safe after out-of-order completion?
Official fact: a committed offset describes the next position from which the application should resume. Kafka's consumer API distinguishes fetched position from committed position, notes that offsets can have gaps, and recommends using the next record or returned next-offset metadata when committing. KafkaConsumer API
In this illustrative partition, records 40, 41 and 42 were delivered. Writes for 40 and 42 are durable, but 41 is still running. The safe next position is 41. Committing 43 would let a restarted consumer skip the unfinished effect for 41. Once all three writes are durable, the next position can advance to 43.
This small Python model expresses the reasoning:
def safe_next(delivered, durable, fetched_next):
for offset in delivered: # delivery order, one partition
if offset not in durable:
return offset
return fetched_next
assert safe_next([40, 41, 42], {40, 42}, 43) == 41
assert safe_next([40, 41, 42], {40, 41, 42}, 43) == 43
assert safe_next([40, 42], {40, 42}, 43) == 43
The gap example matters: do not wait for a nonexistent offset 41. This is a local model, not a production consumer implementation. It assumes the delivered list contains all outstanding records for that partition, no earlier work is omitted, and the next position comes from the client. Real code must retain progress across polls, include commit metadata, bound outstanding work and handle assignment changes.
What happens between a database write and an offset commit?
Official fact: Kafka's consumer documentation describes the failure window between a database insert and committing offsets. Writing first and committing later permits redelivery after a crash. Advancing the commit before processing finishes can skip work. Manual offset control
Here is an original crash ledger for an inventory projection:
| Crash point | Restart behavior | Required defense |
|---|---|---|
| Kafka offset advances before the database transaction commits | Restart may bypass an unapplied update | Advance only after the corresponding durable effect |
| Database transaction commits before Kafka offset advances | The event can be delivered again | Make repeated application harmless |
| Deduplication marker commits separately from the inventory change | Marker and effect can disagree | Commit both within the same database transaction |
For an additive event, give it a stable event ID. In one database transaction, insert that ID into a table with a unique constraint and update the inventory projection only if the insert is new. Replaying the same ID should leave the projection unchanged. A different ID must still apply, even if its timestamp or payload resembles another event.
Our local SQLite check applies two distinct events worth 10 and 20 units, replays the first, and confirms 30 applied units with two event markers. A failure injected between marker insertion and projection update rolls both back. This checks the proposed database invariant; it does not test a live Kafka cluster.
The defense has boundaries. A remote HTTP side effect outside that transaction is not protected by the database marker. Event-ID retention must cover the replay horizon. For mutable state, a duplicate check alone cannot prevent a stale version from replacing current data.
Does producer idempotence mean an exactly-once warehouse?
Official fact: Kafka distinguishes producer delivery from consumer processing. Idempotent production addresses duplicate log writes caused by producer retries. Transactions can coordinate Kafka writes, including consumption progress in Kafka processing designs; guarantees for an external destination require cooperation from that destination. Kafka delivery semantics
Interview reasoning: name the boundary before using “exactly once.” A Kafka-to-Kafka transactional topology and a consumer issuing independent warehouse requests have different failure paths. A read-committed consumer does not automatically make its database writes atomic with Kafka commits.
You can instead explain a specific result: “Events may be delivered more than once, but this database transaction applies each retained event ID once.” Another possible design stores the sink effect and source progress together in the database, then resumes from that durable progress. Both approaches still need an ownership and recovery strategy.
For a deeper transaction discussion, use the separate Kafka transactions interview guide. Keep your initial answer focused on the destination being discussed.
How do you recover when processing stalls or ownership changes?
Official fact: max.poll.interval.ms limits the interval between polling calls for group management. Exceeding it can trigger failure handling and reassignment; static membership changes when reassignment occurs. Configuration also depends on whether the deployment uses the Classic or Consumer rebalance protocol. Consumer configuration and rebalance protocol documentation
An interview recovery design should bound batches and work queues, keep the poll loop responsive, and track durable completion per partition. The Java consumer itself is not thread-safe; worker parallelism requires coordination rather than concurrent calls on one consumer. Consumer threading guidance
When ownership is revoked, decide whether outstanding work can finish within a bounded period or must be cancelled. Stop accepting new work for revoked partitions, avoid advancing beyond unfinished records, and reject stale worker completions using assignment identity. Sink-side version checks or fencing may be necessary when an old worker can still write after a new owner starts.
Increasing a timeout may be reasonable for measured processing latency. It is insufficient evidence that a blocked database connection, unbounded batch or hot partition has been fixed.
What should lag and poison-record tests prove?
Lag needs a stated definition: fetched progress, committed progress and sink-applied progress can differ. Record distance also does not translate directly into seconds of freshness when event rates vary. Inspect both source event time and sink publication time for the business output that matters.
Original test design: inject a slow sink, interrupt processing after its commit, restart consumption, and confirm the expected replay without an extra applied effect. Also test completion order 40, 42, 41; a key-skew workload; and shutdown with outstanding records. Observe the frontier per partition, rather than trusting an aggregate lag chart.
A malformed event creates another decision. Retrying forever can block progress. Skipping it silently can make the dataset incomplete. If policy permits quarantine, durably preserve the event, source position, failure reason and replay route before advancing. Monitor quarantined volume as a data-quality signal. Topic retention determines whether recovery can still read the original input; an offset alone cannot resurrect expired data.
Practice the follow-ups with complete questions
| Question | What to explain |
|---|---|
| Reason About Kafka Consumers and Partitions | Three partitions, five instances, and the ordering cost of scaling |
| Design a Kafka Consumer Service That Groups Trace Events by Trace ID | Key choice, bounded state and completion conditions |
| Explain Kafka Reliability and Offset Recovery | Durable progress and a precise restart position |
| Explain Kafka and Design Bulk Record Processing | Batch completion, partial failure and sink pressure |
| Design Real-Time Analytics Pipeline with Kafka and Flink | End-to-end freshness, state and recovery boundaries |
Return to the partition assignment exercise. Change one assumption at a time and explain which invariant survives. A clear answer should make the restart behavior predictable before proposing more infrastructure.