Estimate project
← All posts

Kafka Without ZooKeeper

Apache Kafka and Apache ZooKeeper went hand in hand for years. This article explains how the new quorum-based Kafka setup works and how it differs from the one that depends on ZooKeeper – but first, let's look at how it all began.

9 min read
Apache KafkaZookeeper

Apache Kafka and Apache ZooKeeper went hand in hand for years. Kafka was responsible for reliable message delivery behind a pub/sub API, while ZooKeeper acted as a distributed store for the cluster’s metadata. ZooKeeper kept track of which replica leads which partition, how partitions are distributed across brokers, consumer offsets, and much more. When Kafka was being built, choosing ZooKeeper took a lot of weight off Kafka developers’ shoulders – they could count on a battle-tested component to keep their messaging system consistent.

This solution has its downsides, though. It makes operations harder: there are two separate infrastructure components to tune, secure and monitor. It also earned Kafka a reputation for being difficult to maintain and heavy on infrastructure. And since Kafka is itself a distributed system, gradually moving all of ZooKeeper’s responsibilities into Kafka was a very attractive idea.

In this article, I’ll walk through how the new quorum-based Kafka setup works and what sets it apart from the previous one – but first, let’s look at how it all began.

Update: Apache Kafka 4.0.0 removed ZooKeeper completely and replaced it with KRaft.

A glimpse of history

The journey officially began around 2019, when KIP-500 was created (KIP stands for Kafka Improvement Proposal). KIP-500 served as an umbrella for several follow-up KIPs, all aiming to build a new internal replacement for Zookeeper. In 2021, release 2.8 brought a beta of the new quorum controller mode based on the KRaft consensus mechanism. Version 3.0 shipped it as a preview. In 2022, Kafka 3.3 declared KRaft production ready (though only for new clusters). The recent 3.5 release offered early access to migrating clusters (with no downtime) to the new ZK-less configuration, and it also deprecated Zookeeper support.

The groundwork for a new, simpler, and better Kafka is already in place, yet the journey isn’t finished. The migration to KRaft will only leave early access in Kafka 3.6, expected in the second half of 2023. Kafka 4.0, planned for 2024, will be the version that drops ZK entirely.
timeline

In 3.5, migrating from ZK to KRaft is still in early access, so it shouldn’t be used in production. For now, KRaft is only ready when you start from scratch. More details are available in KIP-866 / Migration overview or in this tutorial.

The architecture of Kafka with Zookeeper

A Kafka cluster is made up of several nodes known as brokers. A cluster can have as few as one broker, but such a setup only makes sense for testing and development, as it provides no redundancy. Production clusters typically have at least a handful of brokers. Some clusters, however, grow to immense sizes – proof of how well Kafka scales.

Brokers talk to each other over a binary TCP-based protocol. They also exchange messages with the producer and consumer applications that write messages to and read them from topics.

ZooKeeper

Lastly, brokers connect to a cluster of ZooKeeper instances – a so‑called ensemble.

Zookeeper keeps data in data registers known as znodes. ZNodes are organized in a structure resembling a filesystem, and a znode’s name looks like a file path. The zk.connect broker config property tells the broker which ensemble to connect to and under what path. One ZK ensemble can serve several Kafka clusters, as long as they point to different paths. On joining the cluster, a broker creates its own znode, defined by the broker.id property. All brokers in the cluster must use the same address and path in zk.connect, and each broker.id must be unique.

The znodes created by brokers when they join are of a special kind: ephemeral. An ephemeral node is removed from the registry automatically once the connection between ZK and the broker ends. This may happen when the broker leaves the cluster gracefully, but ZK will also end the session if the Kafka instance dies or stops responding (for instance, due to a long GC pause). To show they’re alive, brokers have to send heartbeat signals to ZK. If ZK sees no sign of life from a broker within the time set by zookeeper.session.timeout.ms, it considers the node failed. Each time an ephemeral node is created or deleted, ZK sends out a notification, and other Kafka components subscribe to these notifications to keep an up-to-date view of the cluster state.

Zookeeper is the source of truth for cluster configuration and metadata, holding information about topics, brokers, and ACLs. It is also in charge of appointing a controller broker (covered in the next section).

In the past, ZK had even more responsibilities. Console-based applications (such as console-producer.sh or console-consumer.sh) used ZK as their entry point to the cluster (today, it’s enough to pass a broker address with the –bootstrap-server param).

What’s more, in early Kafka versions (pre 0.9), ZK held information about committed offsets. Nowadays, this information lives in the internal topic __consumer_offsets.

Controller

One broker in the cluster has a special role – it is appointed as the controller. It handles many administrative tasks, such as triggering partition leader elections or reassigning partitions between brokers. It is also in charge of cluster metadata, tracking topics, partitions, replicas, and how they are assigned to brokers. Additionally, it ensures that metadata updates are persisted in ZK and propagated to the other brokers.

At any given time, the cluster may have only one controller. Kafka relies on ZK’s properties to guarantee this. When brokers join the cluster, they race to create an ephemeral znode under the /controller subpath. Because only one znode with a given path can exist, just one broker wins and becomes the new controller. Whenever the controller leaves the cluster, ZK notifies all brokers that they have to compete again to choose a new one.

A broker may sometimes lose its controller status (for example, when a long GC pause kept it from sending heartbeats) without realizing it. It then turns into what’s called a zombie controller. Once a new controller is elected, we could end up with two controllers: the legitimate one and the zombie. To avoid that, each newly elected controller receives a sequential number incremented by one – the controller epoch. Brokers can then simply disregard controller requests that carry a lower epoch.

Before a new controller can begin its work, it has to retrieve the latest metadata state from ZooKeeper and pass it on to the other brokers. In small clusters, this happens fairly quickly. In larger systems, however, fetching can take as long as a few seconds. This is a huge weakness of the Kafka-ZK setup – metadata isn’t kept locally on brokers but has to be read from ZK registers. In large, busy clusters, even a delay of a few seconds can cause serious trouble.

Over the years, the controller election process received many performance improvements, such as asynchronous data transfer from ZK to the broker. Still, it remains a clear bottleneck and scales poorly as the number of partitions grows. Kafka needed a newer, better way of managing cluster metadata.

Kafka with KRaft

The key idea behind the new quorum controller setup is that Kafka is log-based by nature. Metadata changes can be represented as messages stored in a log and then streamed to subscribers.

In KRaft mode, several Kafka instances can be designated as controllers. A node can act only as a broker, only as a controller, or as both at the same time (which is very convenient in smaller clusters). That’s a change from the legacy setup, which had just one controller. Even with multiple controllers, though, only one is active at any moment, while the rest stay on standby. Should the active controller fail, one of the others takes over its duties.

Changes to the cluster’s metadata can be made only by the active controller. It writes the updates to a special internal topic (with a single partition) named __cluster_metadata. All the other controllers then replicate messages from this topic, so each of them holds a nearly up-to-date copy of the data in its local replica. This matters a lot – a new controller no longer needs to pull all the data from Zookeeper. Everything is already in its local log; at most, it has to catch up on a few messages it missed.

Replication

Metadata replication works much like data replication for ordinary Kafka topics. Replicas send a request to the single partition leader (the active controller) with the last offset they’ve seen, and the leader replies with the latest messages. In the happy path, they catch up with the leader quickly.

There are some important differences, however. To begin with, there’s no notion of in-sync replicas (ISR) that could step in when the leader is unavailable. Electing a new leader (who is also the active controller) is always quorum based. As the mode’s name (KRaft) probably suggests, the leader is chosen using the Raft algorithm.

Leader election

The active controller has to send heartbeats to the other controllers at regular intervals. If it doesn’t, they decide it has crashed and begin a new election. Each controller treats itself as a candidate and takes the latest leader epoch it knows of. It increments that value by one and sends it with requests asking the other controllers to vote for it. On receiving a vote request, a controller checks whether it has already seen requests with a later epoch or has already answered a request with the same epoch. If neither is the case, it acknowledges the request. Controllers wait a random period before requesting votes, which keeps all nodes from sending requests simultaneously. Whichever controller collects a quorum of votes becomes the new leader.

Offset committing

Committing offsets for metadata also works differently than for ordinary topics. Kafka has to enforce fsync when appending a new message to the metadata topic. This guarantees that messages are durably stored in the log, which the Raft algorithm requires to remain correct. Each record is tagged with the leader epoch number, which makes it possible to reconcile logs once a new active controller has been elected.

How important is KRaft?

KRaft is a huge step forward for Kafka. It speeds up critical operations, such as recovery after a controller failure, and it removes other Kafka limitations as well. With metadata managed in ZK, creating or deleting a topic takes O(n) (where n is the number of existing topics), since the controller has to load the complete list of topic names from ZooKeeper. With KRaft, these operations take O(1).

Without ZooKeeper, setting up Kafka is far simpler. ZK performance and fine-tuning are no longer our concern. A frequently overlooked benefit of KRaft is a better developer experience. Previously, even a small local single-node Kafka cluster for testing required starting Zookeeper as well. With KRaft, that’s no longer necessary.

Look into the future

Removing the ZooKeeper dependency is among the most eagerly awaited changes in Kafka’s history. Version 4.0, planned for 2024, will end Zookeeper support, and then we’ll be able to say this epic journey has finally come to an end.