Articles

Kafka Connect Interview Questions: Tasks, Offsets, Converters, and Failure Recovery

Practice Kafka Connect interview questions with task status, converter failures, source and sink offsets, safe restarts, and checks for duplicate output.

A Kafka Connect interview answer should explain why data stopped moving before proposing a restart. A connector can report RUNNING while one of its tasks reports FAILED. The next useful evidence is the failed task's trace, effective configuration, and affected record—not another screenshot of the connector's green status.

The safest recovery depends on the failure boundary. A converter mismatch, an unavailable destination, and a deliberate replay require different decisions. Changing offsets before understanding that boundary can turn a contained processing failure into missing or duplicate output.

This guide uses official Apache Kafka 4.3 and Confluent documentation for platform behavior. The status examples and recovery exercises are original practice, not candidate reports or an employer's question list. We ran a local Kafka JsonConverter 4.3.1 test and a separate SQLite replay model. We did not run a Kafka broker, Connect worker, production connector, or external sink.

How Workers, Connectors, and Tasks Fit Together

Official framework behavior: workers execute connectors and tasks; tasks.max is an upper limit, and a connector can create fewer tasks. Converters translate between Connect data and serialized Kafka records, with connector-level overrides available. Exactly-once support depends on the connector and configuration; source support requires distributed mode. Apache Connect user guide

Practice question: you set tasks.max=8, but observe two tasks. Is the deployment broken?

Not from that observation alone. Ask how this connector divides work and how much independent work exists. Then inspect task assignments and throughput. Increasing a configured maximum cannot manufacture parallelism that the connector or input layout does not provide.

Also distinguish the worker process from the connector's logical instance. A reachable REST endpoint establishes that you can contact a worker, not that every assigned task is healthy. A useful status review names the connector, task IDs, assigned workers, states, and observation time.

For an interview, draw three boxes before discussing scaling. Put deployment resources around workers, connector configuration around the integration, and actual work assignments around tasks. This keeps a performance question from becoming an unrelated discussion of adding broker partitions by default.

Diagnose a Running Connector with a Failed Task

Here is an invented, abbreviated status response for an orders sink. The trace is a diagnostic summary, not a verbatim exception captured from a running worker.

{
  "name": "orders-sink",
  "connector": {"state": "RUNNING"},
  "tasks": [
    {"id": 0, "state": "RUNNING"},
    {"id": 1, "state": "FAILED",
     "trace": "DataException during JSON conversion"}
  ],
  "type": "sink"
}

What would you inspect next? Read task 1's complete failure trace and the effective key and value converter settings. Identify which side failed and inspect a safely sampled record's encoding. The abbreviated error does not establish that the destination database rejected anything; conversion may have failed before the sink received the record.

Suppose the record is ordinary JSON, while the effective value converter expects a schema-bearing envelope. The next hypothesis is a representation mismatch. Check whether that format changed recently, whether only one partition contains incompatible records, and whether another connector-level setting overrides the worker default.

Do not infer that task 0 is processing all remaining data simply because it is running. Check assignments and progress for the affected partitions. A partial outage can leave aggregate throughput nonzero while one customer's events remain stuck.

Follow-up: why preserve the original trace before restarting?

A restart can replace the most useful immediate context with a new failure or a transient healthy state. Record the configuration version, affected task, record coordinates when available, and first meaningful cause. Then make a targeted change whose result can be compared with the original evidence.

What the Local Converter Test Actually Shows

We tested the real JsonConverter class from org.apache.kafka:connect-json:4.3.1 on Java 17, using JUnit 5.11.4. The input was:

{"event_id":"e101","amount":20}

One test method checked two configurations. With schemas.enable=true, conversion of that plain object raised DataException, as expected by the test. With schemas.enable=false, conversion returned a value with no Connect schema, and the resulting map contained event_id=e101.

The test finished with one test, zero failures, and zero errors. Catching the expected exception validates the negative case; it does not mean the schema-enabled configuration accepted the record.

Practice interpretation: the bytes can be valid JSON while still violating the configured converter contract. “The JSON parses in my editor” is therefore insufficient evidence that the Connect conversion step should succeed.

Do not turn the result into a universal instruction to disable schemas. A destination connector may require schema information, and existing producers and consumers may rely on a specific representation. Identify the intended contract and fix the inconsistent participant. Changing one setting until this sample passes can create a later failure at the sink boundary.

This was a class-level check, not a worker integration test. It does not establish partition assignment, REST status behavior, offset commits, retries, or successful writes to a database. Those limits matter when explaining what a local reproduction has ruled out.

Can a Dead Letter Queue Handle Every Failure?

Official configuration: Confluent's sink configuration documents error tolerance, error logging, a DLQ topic, and optional error-context headers. A blank DLQ topic means no records are recorded there. These settings do not justify treating every operational failure as a bad record. Sink configuration reference

A concrete connector-specific counterexample is useful: the JDBC sink documentation describes exhausted database connection attempts killing a task without sending records to the DLQ. That behavior belongs to the documented JDBC connector, not an assumed contract for all plugins. JDBC sink overview

Practice question: a teammate proposes errors.tolerance=all to restore throughput. What do you ask?

Ask which failures would be skipped, where skipped records would be retained, who would investigate them, and how completeness would be measured. A green task with silently omitted orders is not a successful business outcome.

Separate malformed-record handling from infrastructure recovery. If credentials expired or the destination cannot accept connections, a DLQ is not a substitute for restoring that dependency. If a record violates an agreed contract, quarantine may be appropriate only with an explicit repair and replay path.

Your answer should name a reconciliation signal. Compare expected event identities with delivered or quarantined identities over a bounded interval. Do not rely only on “DLQ count is low”: an absent or misconfigured DLQ can also have a count of zero.

Restarts, Pauses, Stops, and Offset Changes

Official REST behavior: Confluent documents restarting individual tasks and restarting a connector with task-selection parameters. Pausing differs from stopping; stopping shuts down tasks without deleting the connector. Offset alteration and reset require the connector to be stopped. Source offsets are connector-defined, while sink offsets identify Kafka topic partitions. External offset state may need separate handling. Verify the deployed distribution's API before applying these endpoints. REST reference

For the invented incident, choose an action by its purpose:

Proposed action	Recovery consequence to explain
Restart the failed task after correcting conversion	Re-check the same input contract; a restart alone does not repair incompatible bytes.
Pause processing	Explain what you are containing and verify the observed state; do not assume this satisfies offset-edit requirements.
Stop the connector	Establish the required stopped state before an intentional supported offset change.
Move an offset backward	Identify the replay interval and destination behavior for already-written records.
Move an offset forward	Identify the deliberately omitted records and obtain an explicit completeness decision.

These are editorial decision prompts, not a production command sequence. The right action depends on the connector implementation and what is known about downstream writes.

Follow-up: does a successful REST response prove recovery?

No. Read status again, observe progress on the affected work, and check destination output. An accepted control operation and a healthy data path are different observations. If the same record causes the same failure again, preserve that evidence instead of repeatedly restarting every task.

Explain Which Offset You Mean

An interview answer becomes much clearer when you name the coordinate system. For a source, ask what position the connector records in its external source. For a sink, identify the Kafka topic and partition whose consumption position you are discussing. Do not replace either description with a vague “reset Kafka.”

In a recovery review, keep a before snapshot and the intended after position. Explain why that position corresponds to the records you want to replay or omit. Also check whether the source still retains the required history and whether the sink can distinguish repeat deliveries.

Practice scenario: the destination has already written event e101, but the recovery path delivers it again. If the destination merely appends, another write can create a duplicate. If it applies an appropriate stable-key operation, the repeated delivery may have a different effect. You must inspect the actual sink contract rather than infer it from Connect's presence.

Exactly-once is not a synonym for “I used a connector.” State the boundary of the guarantee, the connector's documented support, and the required operating conditions. Then explain how your chosen destination operation behaves if the recovery assumptions are wrong.

Check Output After a Replay

Our separate SQLite model inserted four deliveries: e100, e101, e101, and e102. It represents an append destination receiving one event twice; it is not an execution of a Kafka sink.

This query found the repeated identity:

SELECT event_id, COUNT(*) AS deliveries
FROM deliveries
GROUP BY event_id
HAVING COUNT(*) > 1;

Observed model output: e101 appeared twice. The table contained four rows but only three distinct event IDs. This is why increasing row count after a restart cannot by itself prove correct recovery.

The next interview question is whether one row per event is the intended contract. Some systems legitimately store multiple versions or delivery attempts. Define the key and expected multiplicity before calling every repetition a defect.

For an event table, compare expected IDs with actual IDs and investigate both missing and repeated records. For a current-state table, also check ordering and versions: a replayed older event might overwrite a newer state even when the row count remains unchanged. Counts are a starting signal, not a complete reconciliation.

Keep the verification interval tied to the incident. An all-time aggregate can hide a small but important gap, while an unbounded replay can make it difficult to identify which records were affected by the repair.

Practice the Recovery Explanation

Practice question	Connect preparation focus
Implement an Idempotent Versioned Database Update	Explain how repeated or older writes affect destination state.
Design a Conversation Log Ingestion Pipeline	Define event identity and ingestion boundaries.
Design Real-Time Analytics Pipeline with Kafka and Flink	Place Connect recovery within the wider pipeline.
Diagnose data quality and pipeline performance issues	Separate throughput restoration from correct output.
Defend a Data Pipeline Architecture and Its Trade-offs	Justify the recovery boundary and its limitations.

Start with the versioned-update exercise, then explain the failed-task case aloud. Your answer should connect the observed failure, one discriminating check, the smallest justified repair, and a concrete destination verification.