35 Kafka Interview Questions and Answers

Apache Kafka is a distributed event streaming platform used to build real-time data pipelines and streaming applications. It is designed for high-throughput, fault-tolerant, and scalable handling of data streams. Common use cases include log aggregation, real-time analytics, event sourcing, and integrating data between different systems.

Kafka is designed for high throughput and scalability, using a distributed, partitioned, and replicated commit log. Unlike traditional message brokers, Kafka stores messages on disk and allows consumers to read messages at their own pace, supporting both real-time and batch processing.

A Kafka topic is a logical channel to which data records are sent by producers and from which consumers read. Topics are partitioned and replicated across brokers, enabling parallelism and fault tolerance.

A partition is a subset of a topic's data. Each partition is an ordered, immutable sequence of records. Partitions allow Kafka to scale horizontally by distributing data and load across multiple brokers, and they enable parallel processing by consumers.

Kafka persists all messages to disk and replicates partitions across multiple brokers. This ensures that data is not lost even if a broker fails, and consumers can replay messages as needed.

A producer is an application that sends (publishes) data to Kafka topics, while a consumer reads (subscribes to) data from topics. Producers and consumers are decoupled, allowing for flexible and scalable data pipelines.

A Kafka broker is a server that stores data and serves client requests. Each broker manages a set of partitions and handles read and write operations for those partitions. In a Kafka cluster, multiple brokers work together to provide scalability and fault tolerance.

A consumer group is a set of consumers that work together to consume data from a topic. Each partition in the topic is assigned to only one consumer in the group, enabling parallel processing and load balancing.

Kafka guarantees message ordering within a partition. All messages written to a partition are stored in the order they arrive, and consumers read them in the same order. However, there is no ordering guarantee across different partitions.

An offset is a unique identifier for each record within a partition. Consumers use offsets to keep track of which messages they have processed. Offsets can be managed automatically by Kafka or manually by the consumer, allowing for flexible message processing and replay.

At-least-once delivery ensures that every message is delivered to consumers at least once, but duplicates are possible. Exactly-once semantics guarantee that each message is processed only once, even in the event of failures. Kafka achieves exactly-once semantics through idempotent producers and transactional APIs, which coordinate message delivery and commit offsets atomically.

Kafka achieves fault tolerance by replicating partitions across multiple brokers. Each partition has one leader and multiple followers. If the leader fails, one of the followers is automatically promoted to leader, ensuring continued availability. Replication ensures that data is not lost even if some brokers go down.

ZooKeeper is used by Kafka to manage cluster metadata, leader election for partitions, and configuration management. It helps coordinate brokers and ensures that only one broker acts as the leader for a partition at any time. However, newer Kafka versions are moving towards removing the dependency on ZooKeeper.

Kafka producers use a partitioner to decide the target partition for each message. By default, if a key is provided, Kafka uses a hash of the key to select the partition, ensuring messages with the same key go to the same partition. If no key is provided, messages are distributed in a round-robin fashion.

Kafka Streams is a client library for building real-time stream processing applications on top of Kafka. It allows for complex transformations, aggregations, and joins of data streams. Kafka Connect, on the other hand, is a tool for integrating Kafka with external systems (like databases or file systems) using connectors, focusing on data movement rather than processing.

Log compaction is a feature in Kafka that retains only the latest value for each key within a topic, removing older records with the same key. This is useful for scenarios like maintaining a changelog or a snapshot of the latest state, such as user profiles or configuration data.

Kafka decouples producers and consumers using persistent storage. If consumers are slow, messages accumulate in the topic partitions on disk. Kafka retains messages for a configurable retention period, allowing consumers to catch up at their own pace without impacting producers.

ISR stands for In-Sync Replica, which is a set of replicas that are fully caught up with the leader for a partition. Only replicas in the ISR are eligible to be promoted to leader in case of failure. This ensures data consistency and durability, as only up-to-date replicas can serve as leaders.

Kafka supports several security features, including SSL/TLS for encryption, SASL for authentication, and Access Control Lists (ACLs) for authorization. These mechanisms help protect data in transit, verify client identities, and restrict access to topics and operations.

Before Kafka 0.9, consumer offsets were stored in ZooKeeper, which could lead to scalability issues. From version 0.9 onwards, offsets are stored in a special Kafka topic (__consumer_offsets), allowing for more scalable, reliable, and performant offset management.

Kafka's replication is partition-based, where each partition is replicated across multiple brokers for fault tolerance. Unlike traditional databases that often use synchronous replication, Kafka uses asynchronous replication for high throughput. Only the leader replica handles reads and writes, while followers replicate data. This design allows Kafka to provide high availability and durability with minimal performance impact, but may risk data loss if the leader fails before followers catch up.

Kafka achieves exactly-once semantics using idempotent producers and transactional APIs. Idempotent producers ensure that retries do not result in duplicate messages, while transactions allow atomic writes to multiple partitions and offset commits. Challenges include coordinating state across producers, brokers, and consumers, handling failures, and ensuring that all components support transactional guarantees.

Kafka stores data in log segments, which are append-only files on disk. Segments are rolled over based on size or time, and old segments are deleted or compacted according to retention policies. Efficient segment management allows for fast sequential writes and reads, minimizes disk seeks, and enables efficient data retention and compaction strategies.

The Kafka controller is a broker elected to manage cluster metadata, partition leadership, and replica assignments. It handles broker failures, triggers leader elections, and updates metadata for producers and consumers. The controller ensures cluster consistency and availability, and its failure triggers a new election to maintain cluster operations.

Kafka guarantees ordering within a partition by ensuring that only the leader replica accepts writes. Replication ensures that followers copy data from the leader. If a leader fails, a new leader is chosen from the in-sync replicas (ISR), which have the latest data. This mechanism maintains consistency and ordering for committed messages, but uncommitted messages may be lost if not replicated before failure.

Increasing partition count allows Kafka to scale horizontally by distributing load across more brokers and enabling more parallelism for producers and consumers. However, too many partitions can increase overhead for metadata management, memory usage, and network traffic. Choosing the right partition count is crucial for balancing scalability, throughput, and resource utilization.

When brokers are added or removed, Kafka triggers a rebalance to redistribute partitions across available brokers. The controller updates partition assignments and notifies producers and consumers. During rebalance, some partitions may become temporarily unavailable, and consumers may experience lag. Proper configuration and monitoring are required to minimize disruption during rebalancing.

Best practices include optimizing producer batch sizes and linger times, tuning broker and OS-level disk and network settings, increasing partition count for parallelism, using SSDs for storage, and configuring appropriate replication and acknowledgment settings. Monitoring and profiling the system help identify bottlenecks and guide further tuning.

Kafka retention policies determine how long messages are kept in topics, based on time (retention.ms) or size (retention.bytes). Proper configuration ensures that storage is used efficiently, old data is purged, and consumers have enough time to process messages. Log compaction can be used for topics where only the latest value per key is needed.

Kafka's transactional API allows producers to send messages to multiple partitions atomically and commit consumer offsets as part of the same transaction. This ensures exactly-once processing semantics. Limitations include increased complexity, potential performance overhead, and the need for all involved components (producers, brokers, consumers) to support transactions.

Large messages can impact broker memory, network bandwidth, and disk I/O, leading to increased latency and risk of out-of-memory errors. Strategies include increasing the maximum message size, using message chunking, compressing messages, or storing payloads externally (e.g., in object storage) and passing references in Kafka messages.

Kafka Connect runs connectors in a distributed cluster, automatically balancing tasks across workers. It monitors connector health, restarts failed tasks, and stores configuration and offsets in Kafka topics for fault tolerance. This design enables scalable, reliable integration with external systems, even in the face of worker failures.

Kafka Streams supports event time processing using watermarks, which track the progress of event time in the stream. Watermarks allow for correct handling of out-of-order events and windowed aggregations. Developers can configure how watermarks are generated and used to trigger computations, ensuring accurate and timely results.

Exposing Kafka over the internet increases risks of unauthorized access, data breaches, and denial-of-service attacks. Security best practices include enabling SSL/TLS encryption, using SASL authentication, configuring ACLs for authorization, restricting network access with firewalls, and monitoring for suspicious activity.

Monitoring involves tracking broker metrics (e.g., throughput, latency, disk usage), consumer lag, partition distribution, and network utilization. Tools like Kafka's JMX metrics, Prometheus, Grafana, and Kafka Manager help visualize and alert on issues. Troubleshooting steps include analyzing logs, checking broker and consumer health, and tuning configurations based on observed bottlenecks.