Estimate project
← All posts

Queues in Kafka: today and tomorrow

See how the KMQ pattern lets you implement some queueing features in Kafka already today.

17 min read
Apache KafkamqperfMessage queuekmqmessaging

Kafka is the top messaging & streaming platform and the de-facto standard for moving large volumes of data. Its core abstraction is the topic, which is split into partitions. Thanks to this partitioning, Kafka topics scale extremely well, with the partition count being the only limit.

Topics can be consumed in a publish-subscribe manner, meaning every client gets every message published to the topic. Such broadcast scenarios have their uses, but more often we want a group of clients to consume messages cooperatively, so that each subscribed client receives a unique subset of messages. Put in JMS terms, we want Kafka to act as a queue rather than a topic.

This is achievable with consumer groups: inside a group, every consumer is given a set of partitions to consume messages from. The consumer group coordinator makes sure each partition is assigned to some client and that no partition is assigned to two clients.

Over the years, this approach has proven itself. Still, it doesn’t handle every use case, which is why a KIP (Kafka Improvement Process) is underway to bring more queueing capabilities to Kafka: KIP-932: Queues for Kafka.

We’ll begin by looking at what consumer groups offer and where they fall short. Next, we’ll go through the new features KIP-932 proposes. Lastly, we’ll show how some queueing features can already be implemented in Kafka today with the KMQ pattern.

If you know consumer groups inside and out, feel free to skip the next section and jump straight to share groups.

Consumer groups

Consumer groups are a thoroughly researched and documented Kafka feature, so we’ll just give a short overview of how they behave and what limits them.
To start with, note that a topic can have many consumer groups, each consuming data independently. This applies to every approach covered in this article; to keep things simple, we’ll always concentrate on how one consumer/share group interacts with one topic (which typically has many partitions).

Partition assignment

As noted above, every consumer belonging to a consumer group gets its own unique set of partitions to consume data from, and it has exclusive access to those partitions.

This reveals the first limitation: the number of members in a consumer group is capped by the number of partitions. Having more members than partitions is pointless, because some members would end up with no partitions to consume from. Although the partition count of an existing topic can be changed, doing so is disruptive and affects how data is partitioned. It’s therefore best to plan ahead – which is never easy.

Whenever a new member joins a consumer group, the group is rebalanced. By default, the algorithm aims to give every member roughly the same number of partitions. Such a rebalance is nonetheless disruptive, even though recent Kafka versions perform it iteratively. If a partition that used to belong to member A is now meant to be consumed by the new member B, we first have to ensure A stops consuming from it and finishes its processing; only after that can B take over. This process is known as fencing.

Message processing

When it comes to the processing itself, every member fetches messages from its assigned topics in the order in which they sit on the partitions. Consumption is therefore partially ordered: if messages M1 and M2 are on the same partition and M1 comes earlier in that partition’s log, the member will receive M1 before M2. Across partitions, however, there is no guarantee about the order of received messages.

After messages are processed, their offset can be committed to Kafka, so that when consumption restarts (because of a crash or a rebalance), delivery resumes from messages not yet processed (not committed). Committing boils down to writing to a special __consumer_offsets topic, which is itself partitioned across Kafka brokers.

In a consumer group, though, messages can only be committed up to an offset: all we can state is that every message with an offset lower than a given one counts as committed. This solution is simple and fast, but limited – messages can’t be acknowledged one at a time, only in batches up to an offset.

One could, of course, process messages individually and commit each offset before moving on to the next message. The performance cost of that would be severe and wide-ranging, though, so this approach is very rarely (if ever) combined with Kafka. It simply runs counter to how consumer groups and the offset commit mechanism are designed.

consumer group

Other characteristics

You also can’t ask for a single message to be redelivered or flag a message as unprocessable. The consumption pattern that consumer groups support can also cause trouble when messages take very different amounts of time to process. This is called head-of-line blocking. One message that takes much longer to process than the rest may hold up all later messages from that partition: in a typical consumer flow, we first fetch a batch of messages, then process them (possibly in parallel), and finally commit their offsets once every one of them has been processed.

Following the receive-process-commit offsets algorithm gives us at-least-once delivery semantics: the process may crash between processing and committing the offsets. If that happens, the messages whose offsets weren’t committed will be delivered and processed once more after the partition is reassigned.

Swapping the steps in that flow – receive-commit offsets-process – gives us at-most-one instead.
Performance-wise, consumer groups are tough to beat: messages are fetched in batches, and offsets are committed in batches too, possibly in the background.

Consumer Groups summary

  • every member of a consumer group gets an exclusive set of partitions to consume data from
  • the number of consumers – and therefore how parallel data processing can be – is capped by the number of partitions
  • messages that consumers receive are partially ordered within each assigned partition
  • messages can only be committed “all up to an offset”, in batches
  • individual acknowledgment and message redelivery are not possible
  • when message processing times vary, head-of-line blocking may affect the latency of processing messages within a partition
  • messages are processed with at-least-once or at-most-once semantics
  • communication overhead is very low: offsets only need to be committed from time to time
  • performance is very high

Queues for Kafka: share groups

To remove some of the consumer-group limitations described above, KIP-932: Queues For Kafka proposes adding a share group abstraction to Kafka. Share groups, like consumer groups, enable messages from a Kafka topic to be consumed cooperatively. There are, however, several significant differences.

Partition assignment

When a member joins a share group, it gets assigned a number of share-partitions from which it can fetch messages for processing. In contrast to consumer groups, this assignment isn’t exclusive: many members may be assigned the same partition and consume data from it. That removes the cap that the partition count places on the number of group members. At the same time, it brings a far harder problem: making sure each member gets different messages to process.

Before we get into that, let’s take a closer look at how a member joins a share group. A newly joining member sends a heartbeat to the group’s coordinator, which lives on one of the brokers – the partition leader for the consumer offsets topic. That broker runs the partition-assignment algorithm, and in reply to the heartbeat the member gets the share-partitions it should consume. Assignments can change, and other members find out which partitions have been added to or taken away from them with their next heartbeats.

Fencing isn’t needed, since several members can consume messages from one partition. So even if, for a short time, more members consume a partition than the assigning algorithm intended, it makes no difference. This makes rebalancing simpler, with no “stop the world” fencing events.

Message processing

At this point the group coordinator’s job is done. The members then send a fetch-messages request to every share-partition assigned to them. Each share-partition keeps track of the state of the messages it holds; a message can be:

  • available for consumption
  • acquired, but not yet acknowledged (in other words, currently being processed)
  • acknowledged
  • rejected

On receiving a new fetch-messages request from a member, the share-partition checks its internal state and decides which messages to deliver. Where it can, the share-partition tries to deliver messages in batches – so that acknowledgments can be batched too (by passing offset ranges rather than individual offsets). This matters a lot for performance, naturally.

Out of the messages available for consumption, the share-partition picks a batch and marks it as acquired. These messages are then handed to the member for processing and stay in the acquired state for a set period of time. If they aren’t marked as processed before it expires, they go back to the available state. The model is similar to, for example, SQS or GCP’s Pub/Sub, where message processing has to be acknowledged within a given time window, or else the messages are delivered again.

share group

Other characteristics

Every message (or range of messages) has a delivery counter attached. The counter goes up each time message acquisition times out or the member explicitly releases a message (signalling that it should be redelivered). Once the number of delivery attempts passes a set threshold, the message becomes rejected and no further delivery is attempted. There’s no DLQ – at least not yet, although it’s mentioned as a possible future addition. Also, the delivery counter should be regarded only as an approximation, because broker failures may cause it to differ from the actual number of delivery and processing attempts.

Kafka’s share group implementation limits both how long processing a message may take (as explained above) and how many messages can be in flight, i.e. made available for delivery. This is governed by the share partition start and end offsets (SPSO and SPEO). Besides the message states (acknowledged/ rejected/ delivery count), the SPSO also forms part of a share partition’s state.

Since many consumers can get messages from any partition and messages can be acknowledged one by one, there are no ordering guarantees whatsoever. When a fetch request comes in, any available message may be handed to the consumer. Gaps can appear because some messages are being redelivered or have already been acknowledged; on top of that, different consumers get different batches of available messages.

Coordinators & persistence

A share partition’s state has to be persisted so that it survives broker crashes. This happens on a dedicated topic and is managed by a share coordinator. In the same way that group coordinator duties are spread across the brokers, share coordinator duties are spread out too. The share coordinator persists message states and group epochs in a dedicated, heavily partitioned topic. It runs alongside the leader of the __share_group_state topic partition that stores the data for a given share-partition. Which broker that is can be found via a Kafka protocol for locating coordinators (the same mechanism serves the new consumer group implementation, group coordinators and share coordinators). You can read more about it in KIP-848: Next Generation of Consumer Rebalance Protocol.

The state persisted in the __share_group_state topic is deliberately selected and kept minimal. To begin with, message acquisition isn’t persisted; only once messages are acknowledged or rejected does this enter the persisted state. This reduces inter-broker traffic as well as the volume of data to persist. On the other hand, if a broker crashes, some messages may be redelivered even though their acquisition time window hasn’t expired. Consumer groups can behave the same way: messages are “acquired” when delivered to a consumer, and offsets are persisted only after messages are acknowledged. Although unlikely, two consumers could end up processing the same messages at the same time.

Share group assignments aren’t persisted either – only the epochs are, which guard against zombie group coordinators and let members check whether the assignments they get are current. These optimizations are meant to cut down the amount of data kept in the __share_group_state topic.

Every share coordinator keeps an in-memory view of the state of the share-partitions whose data it stores, and from time to time writes snapshots to the underlying storage. Once snapshots for all share-partitions have been persisted, any earlier state updates can be removed; this compaction is also handled by the share coordinator’s logic.

Share groups summary

To sum up, the Queues for Kafka KIP brings in a new concept – share groups – which let any number of consumers consume messages cooperatively. Three new stateful components run in a distributed fashion across Kafka brokers: the group coordinator, the share coordinator and share-partitions. Between them, they keep in-memory views of each share-partition’s state (which messages can be delivered, which have been acknowledged and so on) and of how members are assigned to partitions. This state is persisted on a new __share_group_state topic and the already existing __consumer_offsets one.

We’ve talked about consumer groups’ head-of-line blocking problem. Share groups can potentially solve it, provided each partition is consumed by more than one consumer. For that, we’d need far more consumers than partitions: by default, the broker-side partition assignor gives each consumer as few partitions as possible. Custom assignors can be supplied, though, so that, for example, every partition is guaranteed to be consumed by several consumers. If messages delivered in a batch are then processed in parallel instead of one after another, head-of-line blocking could be resolved.

Performance remains unknown (the design isn’t final yet – the feature is currently slated for Kafka 4.0), or at least we haven’t come across any benchmarks. Still, we shouldn’t anticipate problems: messages are fetched in batches straight from brokers, and acknowledgement can (and should!) be batched as well. One thing that could affect latency is that acknowledging messages involves up to 3 network hops (consumer -> share-partition -> share coordinator -> replicas), versus 2 network hops in consumer groups (consumer -> consumer offsets -> replicas). As long as this is done in batches, though, the extra cost should amortise easily.

Let’s go back over the characteristics listed in the consumer groups summary:

  • every member of a share group gets a non-exclusive set of partitions to consume data from
  • how parallel data processing can be is not capped by the number of partitions
  • received messages come in no particular order
  • messages can be acknowledged, rejected or released for re-delivery one by one
  • under the hood, Kafka tries to batch both message delivery and acknowledgment
  • the head-of-line blocking problem can potentially be solved
  • messages can be processed with at-least-once or at-most-once semantics
  • communication overhead is low: messages have to be acknowledged (possibly in batches) and share-partition state has to be persisted
  • performance is not yet known but should be high

KMQ

KMQ is a pattern that implements individual message acknowledgements on top of the functionality offered by Kafka’s consumer groups, so it works with any Kafka version. You may spot some similarities between the designs of KMQ and share groups. KMQ is, however, much simpler to implement and deploy, which also means it tackles only some of the weaknesses of consumer groups. KMQ lives entirely on the consumer side, unlike share groups, where substantial logic runs within the brokers.

In the same way that share groups depend on extra components (group and share coordinators, share-partitions) and extra topics for persisting meta-data (__consumer_offsets and __share_group_state), KMQ relies on one extra component – the redelivery tracker – and one extra topic for meta-data, the markers topic.

Message processing

Consuming messages from a topic involves a few additional steps on the consumer side. This resembles using the dedicated KafkaShareConsumer in share groups and comes as part of the published KMQ package. That said, as far as we know, many people treat KMQ as a pattern and just copy the message-receiving logic into their own code base, tweaking it as they need.

Consumers of a KMQ topic should make up a consumer group, so – exactly as with consumer groups – processing parallelism is capped by the number of partitions of the topic holding the messages. After receiving a batch of messages from its assigned partitions, the consumer should write start markers for every received message (comparable to the acquired state in shared groups). Here lies the key difference from consumer groups: once the start markers have been written, the consumer should commit the offsets of the consumed topic and only afterwards process the messages. When processing is done, the consumer should write end markers, which corresponds to acknowledging the messages in share groups.

kmq

Message redelivery

To redeliver messages correctly, an extra component – the redelivery tracker – is required. It should be deployed like any other Kafka application, consuming the markers topic cooperatively as part of a consumer group (a different one from the group above). Much as the share coordinator’s duties are spread over broker nodes in share groups, here the redelivery tracker’s duties are spread over consumer nodes. When a redelivery tracker node fails, rebalancing the consumer group hands its partitions to the remaining working redelivery tracker instances.

The redelivery tracker consumes the start & end markers from the markers topic and updates its internal state, which mirrors the state of message delivery. KMQ relies on a comparable timeout mechanism, redelivering messages when no end marker is written within the configured message timeout. Redelivering a message simply means writing it to its topic again, and the delivery counter in the message metadata is updated at the same moment. Once the delivery counter goes above the configured threshold, the message is written to a DLQ instead.

The redelivery tracker commits offsets in the markers topic using the highest offset. Every start marker has a matching end marker (these are written for timed-out messages too). Thanks to that, the markers topic can be given a retention period of, say, twice the message timeout, leaving the redelivery tracker plenty of time to redeliver any messages that timed out.

Finally, regarding performance: writing the start/end markers does add overhead, but just like in the other solutions it can be batched, spreading the cost so that it ends up fairly low per message. As in consumer groups, “acknowledging” messages – i.e. writing the end markers – takes two network hops (consumer -> markers topic -> replicas). A while ago we benchmarked KMQ against consumer groups and got comparable throughput results.

KMQ summary

  • every consumer gets an exclusive set of partitions to consume data from
  • how parallel data processing can be is capped by the number of partitions
  • received messages come in no particular order
  • messages can be acknowledged one by one; rejection and release aren’t implemented, but adding them would be trivial
  • markers are written in batches through Kafka’s standard write batching mechanisms
  • head-of-line blocking may be a problem, exactly as with consumer groups
  • messages are processed with at-least-once semantics
  • communication overhead is medium: a start and an end marker must be written for every message, and offsets must be committed periodically
  • performance is high

In comparison

We’ve looked closely at three ways in which several client consumers can cooperatively consume messages from a Kafka topic. Here’s a summary of what each offers:

Consumer groups Share groups KMQ pattern
Availability Part of Kafka out of the box May arrive in future Kafka versions A pattern you implement on top of any Kafka version
Acknowledgments Only every message up to a given offset Per-message acknowledgments Per-message acknowledgments
Number of consumers (processing parallelism) Capped by the number of partitions No cap Capped by the number of partitions
Ordering Messages ordered within each partition No ordering No ordering
Head-of-line blocking Can occur Avoidable Can occur
Delivery semantics At-least-once / at-most-once At-least-once / at-most-once At-least-once
Communication overhead Very low Low Medium
Performance Very high Likely high High

Building on its simple foundations, Kafka keeps adding new functionality and addressing the needs of all kinds of use cases. Fundamentally, Kafka ensures data durability by replicating topics across multiple brokers and offers scalability by splitting topics into a number of partitions. This design has its limits, but well-designed applications can often work around them.

Message queueing illustrates this well: in many cases, consumer groups are all you need. For more demanding use cases, patterns like KMQ may be the solution. And to solve the queueing problem in Kafka completely, there’s a proposal to extend its functionality with share groups.

We hope this overview has been useful and helps you pick the right way to consume messages in Kafka.