Estimate project
← All posts

Running Apache Kafka in the Cloud – what are the options?

Planning to use Apache Kafka in your microservices? Running brokers on plain virtual machines or on-premises tends to be complicated. Consider a cloud-based offering instead.

7 min read
CloudApache KafkaAWSKubernetesGoogle Cloud Platform

Introduction

Many companies rely on Apache Kafka, a powerful technology applied across a wide range of use cases and applications. In this post we concentrate on setting up a Kafka cluster. If you plan to build event-driven microservices on Kafka, configuring brokers on bare VMs or on-premises can turn out to be far too complex, particularly when you lack hands-on experience. Below we look at the easier routes offered by various cloud platforms.

Kubernetes Operator

Before we get to the offerings tied to particular clouds, let’s begin with Kubernetes. There is a good chance your organization already runs it. In that case, a sensible choice might be to deploy a Kafka Cluster on K8S with one of the existing Kubernetes Operators. This could be the least expensive route, but remember that maintenance & monitoring then fall on you, which increases the overall cost.

Strimzi

Strimzi is most likely the best-known Kafka operator. Beyond running brokers, it supports Kafka Connect, Mirror Maker, Exporter (for Metrics) and Kafka Bridge (HTTP API). It can even handle topic management automatically. We have relied on Strimzi in several of our projects and it performed rather well.

Confluent operator

Alternatively, you can use the Kubernetes Operator that Confluent has published. It works only with the commercial Confluent Platform. On the other hand, it covers more products from the Kafka ecosystem than Strimzi does - for example Schema Registry and ksqlDB.

Koperator

There is yet another operator, not as widely known but quite promising. The Koperator, previously called Banzai Cloud Kafka Operator, now belongs to Cisco. With it you can deploy a Kafka cluster along with Cruise Control and Prometheus. The Open Source edition comes with a reduced feature set, while the enterprise edition lets you use Kafka Connect and ksqlDB.

Cloud-specific

We frequently notice that business clients would rather not run a cluster of their own. It is a complex task: you have to hire people with the right experience and keep the infrastructure going. So they opt for a “Kafka as a service” kind of product instead. Let’s go through several leading cloud providers and check what each one offers.

Amazon Web Services (AWS)

Amazon Web Services provide several products suitable for data streaming or passing events. The key ones are AWS SQS and Amazon Kinesis. If you look under the hood of Kinesis, it resembles Kafka and some of its concepts quite closely. Still, you cannot simply swap one for the other, as the APIs are not compatible. Luckily, managed Kafka instances are available as another option.

Amazon Managed Streaming for Apache Kafka (Amazon MSK)

Amazon MSK gives you a managed Kafka cluster. You are billed for broker instances and storage (the ZooKeeper service is not charged). It provides high availability through automatic multi-az replication. Data is encrypted both at rest and in transit. The service comes with a 99.9% uptime SLA, but it applies only to certain scenarios, e.g. failures resulting from bugs in Apache Kafka itself are excluded. Some describe MSK as only partially managed. It is up to you to choose the sizes of Kafka server nodes, track performance and scale as needed. AWS has published a rather extensive guide on best practices for sizing Kafka clusters on MSK.

Amazon MSK Serverless

MSK Serverless aims to turn MSK into a fully managed service. Billing is hourly and covers the cluster, the number of partitions, storage, and inbound and outbound data transfers. As with most serverless offerings, this will pay off in some use cases and cost more in others. MSK Serverless is not covered by the 99.9% SLA.

MSK Connect

Amazon MSK Connect delivers Kafka Connect as a managed service. You pay hourly for connector usage, based on the number of workers. It works with MSK, but it is not limited to it - other Apache Kafka clusters are supported too.

Others

The AWS portfolio lacks Kafka Streams and the standard Schema Registry, though it provides some substitutes. To process MSK messages and stream data you can use Amazon Kinesis Data Analytics, and you can also rely on AWS Glue Schema Registry. On top of that, it connects with many other AWS services.

Google Cloud Platform (GCP)

GCP has no managed Kafka offering. For messaging, its go-to products are Pub/Sub and Pub/Sub Lite. The idea behind the Lite version resembles Kafka, and interestingly, it exposes a “Kafka API”.

Pub/Sub Lite with Apache Kafka-like API

You can combine the Kafka client library with the Google Pub/Sub Lite Kafka Shim Client.
Sadly, this comes with certain restrictions:

  • transactions are not supported,
  • you can produce or consume messages from only one topic at a time,
  • sending messages to a chosen partition is not possible.

Under the hood, it talks to Pub/Sub Lite services over gRPC.

Pricing is based on reserved capacity - throughput, storage and egress. Depending on the topic type, uptime varies from 99.5% to 99.95%. The service is compliant with HIPAA and SOC2.

Azure

Apache Kafka on HDInsight architecture

Azure HDInsight is a product for running Apache Hadoop, Apache Spark and other big data systems. You can use it for Apache Kafka too. Billing covers the provisioned cluster (the price depends on how many nodes you have and of what type). Data is encrypted at rest and the service guarantees 99.9% uptime. Monitoring is done with Azure monitor, in a way rather similar to Amazon MSK.

Azure Event Hubs with Apache Kafka API

This approach resembles GCP Pub/Sub Lite. You can work with Event Hubs through Kafka Client libraries. It has certain limitations, for instance log compaction is not supported. Instead of Kafka Streams, Azure suggests one of its stream processing services that can simply be used with Event Hub. Pricing is based on provisioned capacity and the selected plan. Uptime varies by plan, from 99.95% to 99.99%.

Externally Hosted-services

Besides services specific to a given cloud, third-party companies will run and manage Kafka and related products in whichever cloud you prefer, in some cases even within your own cloud account.

Confluent Cloud

Confluent Cloud offers the widest range of well-known Kafka-related services. It supports Connect (Confluent connectors included), Schema Registry, and also ksqlDB. It additionally provides several proprietary solutions - such as Stream designer or Stream Governance - which can serve as a good foundation for an event streaming platform.

Pricing is determined by several factors. You can pick one of 3 plans:

  • Basic,
  • Standard,
  • Dedicated.

They differ in features, prices and compliance certifications. The top plan supports ISO 27011, SOC 3, HIPAA and PCI, and provides multi-az deployment with 99,99% uptime, data encryption and infinite storage. It can run on AWS, Azure or GCP. The price varies with the plan, the cloud provider, the volume of data in & out, storage and other factors. There is no node sizing involved - this is a fully-managed option, somewhat like MSK Serverless. Bear in mind, though, that given the MSK limitations and SLA exclusions, Confluent comes out considerably better in this respect.

Instaclustr

Instaclustr provides a range of open source projects as managed services - e.g. Kafka, Kafka Connect and Schema Registry. It works with AWS, Azure and GCP, and in addition with IBM Cloud and DigitalOcean. It is compliant with PCI-DSS, SOC2 and HIPAA. Notably, it comes with a 99,999% SLA and can run within your own cloud account. And pricing? It is determined by the plan (Developer or Production) and by the number and size of nodes (for both Kafka and Connect).

Interestingly, Instaclustr forked Schema Registry before its license was changed. Today they also support Karapace - a Schema registry replacement built by Aiven.

Aiven

Aiven is fairly close to Instaclustr. It provides Kafka, Kafka Connect and Karaspace. It works with AWS, Azure, GCP, DigitalOcean and UpCloud. There are 3 pricing plans to choose from: Startup, Business and Premium. Only premium can be deployed in your own cloud account. Each plan has a different maximum storage, and within each of them you can pick from various CPU & RAM configurations per VM.

Summary

Apache Kafka is turning into the de-facto standard for event-driven and data streaming architectures, which is why every major cloud provider has Kafka-related services in its portfolio. These differ in pricing models, SLA terms and features. Because of licensing restrictions, some of the projects cannot be provided as SaaS by anyone other than Confluent. As a result, alternatives are being built, or integrations with other comparable existing services.

Which option should you pick for your next project? There is no simple answer. It depends! It depends on the SLA, pricing and security requirements - which certifications you need, and whether you require a 100% automatically managed environment or can accept some maintenance. It depends on the stage of the project and on the performance requirements. Many factors have to be weighed before you decide. Good luck!

Would you like to introduce Apache Kafka in your company or scale the solutions you already have? We are a Confluent Premium Partner with numerous certified experts on board, and we will gladly help!