Kafka and AI stroll into a bar...
Find out how Kafka, AI, and the Model Context Protocol (MCP) fit together. See how to configure mcp-confluent with Claude Code, manage clusters, and get past the pain points of the CLI.
Administering Kafka clusters makes up a large share of my daily work. There are some quite decent UI tools, such as AKHQ or Kafkdrop, but let’s face it – in many situations, including working straight on the brokers/controllers, the CLI tools do the job best.
The CLI tools are fairly stable and you know how to work with them, but who ACTUALLY remembers every one of these --very-long-switches?
At times they defy common sense, for example:
- I can provide a (comma separated) list of bootstrap servers, so surely the option is
--bootstrap-servers? Nope, it’s--bootstrap-server. - You’d like to give
kafka-topicsthe path to the client configuration, so it must be--client-config? Nope, it’s--command-config. - Do the console producer and consumer also take
--command-config? Nope. They use--producer.configand--consumer.config, respectively. Am I the only one who finds switching from the--dash-conventionto the--dot.conventioncompletely illogical?
It’s August 2026, and AI is quite literally everywhere. So the obvious question is: must I keep wrestling with all of this by hand? Is there a tool that could handle it all for me? On top of that, I’m an old-school admin, so I’d rather use the terminal than flashy, polished UIs.
Let’s experiment with mcp-confluent, which is meant to solve precisely this problem – managing Kafka clusters by talking to computers the way we talk to other people.
So what exactly is MCP?
Before answering that, let’s clarify what an LLM is. An LLM (Large Language Model) is a body of static knowledge (apologies to all AI experts for simplifying this much). LLMs do a great job with questions such as “How..” or “What is..”. But what about “Do something”?
MCP (Model Context Protocol) is an open standard which, put simply, lets LLMs make use of external systems.
With MCP servers, you can:
- reach external data,
- invoke tools,
- carry out actions on the user’s behalf,
- operate on live systems instead of relying solely on static knowledge.
MCP usually exposes three types of capabilities:
- tools, i.e. actions the assistant is able to perform;
- resources, i.e. data the assistant is able to access;
- prompts, i.e. predefined templates for tasks or workflows.
Mcp-confluent overview
I chose to experiment with a tool built by Confluent (bear in mind that several Kafka-related MCP servers exist). mcp-confluent naturally works with Kafka, but it spans the entire Confluent ecosystem – it can handle Schema Registry, Flink, Kafka Connect, etc. (depending on how Kafka is deployed; more about that later).
Configuration
The documentation says setting it up is quite simple – just run init:
npx @confluentinc/mcp-confluent --init-config
And then edit the file.
Honestly, though, it reminded me of the instructions for drawing the owl:
The problem – mcp-confluent is heavily focused on Confluent Cloud, and it took me some time to set it up correctly for a generic Apache Kafka cluster.
Below is a sample configuration that connects to two Kafka clusters using common authentication methods: mtls and SCRAM:
server:
transports: [stdio]
log_level: "${LOG_LEVEL:-info}"
connections:
mtls:
type: direct
kafka:
bootstrap_servers: "${BOOTSTRAP_SERVERS:-localhost:29092,localhost:29093,localhost:29094}"
# No auth block: the client certificate is the identity (mTLS).
extra_properties:
security.protocol: "SSL"
ssl.ca.location: "${SSL_CA_LOCATION:-./clusters/mtls/secrets/ca.crt}"
ssl.certificate.location: "${SSL_CERT_LOCATION:-./clusters/mtls/secrets/client.crt}"
ssl.key.location: "${SSL_KEY_LOCATION:-./clusters/mtls/secrets/client.key}"
ssl.endpoint.identification.algorithm: "https"
schema_registry:
endpoint: "${SCHEMA_REGISTRY_ENDPOINT:-https://localhost:8082}"
scram:
type: direct
kafka:
bootstrap_servers: "${BOOTSTRAP_SERVERS:-localhost:19092,localhost:19093,localhost:19094}"
# key -> sasl.username, secret -> sasl.password (these are intentionally NOT
# allowed in extra_properties; they belong in the auth block).
auth:
type: api_key
key: "${KAFKA_USERNAME:-mcp}"
secret: "${KAFKA_PASSWORD:-mcp-secret}"
extra_properties:
security.protocol: "SASL_PLAINTEXT"
sasl.mechanism: "SCRAM-SHA-512"
schema_registry:
endpoint: "${SCHEMA_REGISTRY_ENDPOINT:-http://localhost:8081}"
One setting is particularly important: if you plan to use it against production clusters, you may want to switch it to read-only mode. To do so, just add read_only: true at the connection level.
Included tools
An MCP is only as clever as the tools it provides. mcp-confluent turns on a toolset based on the Kafka deployment type: it detects whether you’re working with Confluent Cloud or with a local deployment.
The complete toolset is only available for a CC deployment; for a local deployment, it’s restricted to Kafka and Schema Registry.
Either way, you also get access to the documentation, which makes it a very effective search engine.
Here’s the full list of tools:
- Kafka
- Flink SQL
- Flink Catalog
- Flink Diagnostics
- Connectors
- Schema Registry
- Catalog and Tags
- Organizations, Environments and Clusters
- Tableflow
- Tableflow Catalog
- Metrics
- Billing
How smart the MCP is depends entirely on these tools: which tools are available and what each of them can do. Let’s see what mcp-confluent is capable of.
Time to play
Disclaimer: I was using mcp-confluent v1.5.0. Newer versions may behave differently.
For my playground, I set up 2 Kafka clusters with fairly standard authentication setups: SASL/SCRAM and mTLS. I also created several topics, each relying on Schema Registry with a different format (JSON, Avro, and Protobuf).
Integrating with clients
Setting up an MCP without an LLM doesn’t make a lot of sense. Detailed instructions for configuring each client are in the README file. I went with Claude Code and just ran this command:
claude mcp add confluent -- npx -y @confluentinc/mcp-confluent --config /path/to/config.yaml
Note: the config refers to certificates by relative path, so run the command from the directory containing the config.
Basics
Let’s begin with something simple – which kafka clusters are available?
As you can see, CC immediately figures out how to locate the cluster, describes it, and even suggests what to do next. Fine, let’s go ahead.
That’s reasonable. What does a specific topic on one of the clusters look like in detail?
Not quite what I was after, but at least it explained what went wrong. Let’s add NODE_EXTRA_CA_CERTS.
Viewing the detailed topic configuration relies on a REST endpoint, which a plain Apache Kafka installation doesn’t provide. That’s a major drawback.
In the same way, we can list all topics:
Producing and consuming
Let’s try producing and consuming a few messages.
My first attempt was to produce to “payments”, which uses a PROTOBUF schema, and it failed:
This is therefore a real bug that nobody has reported yet: any PROTOBUF subject that was registered as .proto text via the Schema Registry REST API – the usual, spec-compliant way – can’t be produced to through the use-latest path. It works only when the schema was first registered by this SDK’s own serializer as a base64 FileDescriptorProto. Your seed/register-schemas.sh follows the standard approach, and that’s exactly why it fails.
Producing to a topic with an AVRO schema went through without any problems:
Consuming works too:
What’s really nice is the explanation of how ordering works in Kafka – great for learning.
Consumer groups
Now let’s try listing consumer groups (I had the console-consumer running in another shell).
Fun fact: when I asked to reset the offset for CG1 to the start of the log, Claude Code discovered that the MCP server can’t do that, but it was clever enough to notice that Kafka was running in a docker container, so it ran the native kafka-consumer-groups CLI tool instead (and it worked!).
Takeaways
Is mcp-confluent a good fit for daily work with Kafka? Frankly, I’m not convinced. Sure, it handled the basics, but every command meant waiting tens of seconds. It could serve well as a debugging tool while building an application that relies on Kafka, but I’ve worked extensively on such applications and managed without an MCP server – the AI just used the CLI tools (much like the offset change in this session).
A major plus of this approach is the tool for reading the documentation, but AI tools can get the same result without MCP simply by browsing the web.
As for me, I’ll most likely stick with the old-school CLI :).