What Kafka Lag Monitoring Quietly Gets Wrong
At 3 AM your alert goes off: "Consumer group orders-processor is lagging by 50,000 messages." Nervous, you rush to your laptop. Is that 10 seconds, 10 minutes, or 10 hours of delay? The alert won't tell you. The dashboard still reads 50,000 messages. So what does that number really mean for your users?
At 3 AM your alert goes off: “Consumer group orders-processor is lagging by 50,000 messages.” Nervous, you rush to your laptop. Is that 10 seconds, 10 minutes, or 10 hours of delay? The alert won’t tell you. You open the dashboard - it still reads 50,000 messages.
So what does that number really mean for your users?
The metric everybody watches - offset lag - says next to nothing about how healthy a consumer really is. Depending on the topic’s throughput, a lag of 1,000 messages might mean a delay of 1 second or of 1 day. And yet that is the figure splashed across dashboards all over the place.
What really matters is time lag - how many seconds your consumer is actually behind. Most monitoring tools, however, either report it inaccurately or not at all.
Below I explain why, and what you can do about it.
The offset lag mirage
First, the basics. Offset lag is computed as:
offset_lag = latest_offset - committed_offset
Here:
- latest_offset is the position at which the next message will be written (high watermark)
- committed_offset marks how far your consumer has processed
Easy, isn’t it? Take two numbers, subtract one from the other, and you’re done.
That ease is precisely why every monitoring tool displays it. The Kafka Admin API hands you both numbers straight away - nothing to fetch, no timestamps to parse, only metadata.
And this is exactly where it falls apart.
Offset lag has no connection whatsoever to time. Look at these two scenarios:
The same kind of metric, yet wildly different severity. That 3 AM alert about 50,000 messages? On a high-traffic topic, your on-call engineer may be losing sleep for no reason. On a quiet topic, even 50 messages might mean the consumer has been down for days.
The idle producer pitfall
It gets worse. What happens once a producer stops sending messages?
Offset lag stays flat. The committed offset does not change, since there is nothing new to consume. The latest offset does not change either, since nothing is being produced. The difference? It does not change at all.
Meanwhile, the real delay keeps climbing. If the last message was produced an hour ago and your consumer is still parked on that offset, you are an hour behind in real time - even though offset lag claims all is well.
I have watched teams get caught by precisely this situation. A low-traffic topic handling once-a-day batch jobs reported “stable lag” for a whole week. As it turned out, the consumer had crashed, but because the producer sent messages only once a day, the lag figure hardly changed. Users noticed the issue when their daily reports stopped showing up.
Why getting time lag right is hard
If time lag matters so much, why don’t all tools display it?
Because computing it correctly is costly and fiddly.
Time lag is defined like this:
time_lag = current_time - timestamp_of_message_at_committed_offset
To obtain that timestamp, you have to actually read the message sitting at the committed offset. In other words:
- Set up a consumer
- Seek to the committed offset
- Fetch the message
- Parse its timestamp
- Repeat this for every partition that has lag
On a cluster with hundreds of consumer groups spread over thousands of partitions, that quickly becomes a lot of work. Most tools skip it altogether.
The interpolation shortcut (and where it breaks)
Some tools try a smarter route. The widely used kafka-lag-exporter (written in Scala) gradually builds a lookup table of (offset, timestamp) pairs. To estimate the timestamp of a given offset, it interpolates between the points it already knows.
That sounds sensible. Here is why it fails:
- Cold start problem: The lookup table begins empty. Until enough data points have been collected, the metric has no meaningful value.
- Idle producer edge case: Once production stops, the lookup table collapses repeated offsets into a flat segment. Interpolating inside a flat segment leads to division by zero (dy = 0 in the prediction formula), so time lag comes out as NaN.
- Variable throughput: Linear interpolation assumes the message rate between data points is constant. Bursts that happen within a single poll interval (default 30s) produce inaccurate time estimates, because the real, non-linear production pattern is not captured.
Whenever interpolation fails (cold start, too little data), the system reports time lag as NaN. Dashboards show this as gaps in the data rather than as zero. The offset lag metric (which does not depend on interpolation) stays accurate even when time lag cannot be computed.
Doing it properly: sampling timestamps directly
Accuracy has no shortcut. To get real time lag, you have to read the actual timestamps of messages.
This is how klag-exporter does it:
For every partition where lag > 0:
- Set up a consumer and seek to the committed offset
- Fetch a single message
- Read its timestamp
- Compute:
time_lag = now - message_timestamp - Cache the result so Kafka is not hammered
The crucial observation is that timestamps are needed only for partitions that actually lag. A consumer that has caught up (lag = 0) needs no timestamp - we already know it is up to date.
For lagging partitions, the fetched timestamp is cached with a TTL. As long as the consumer has not moved (the committed offset is unchanged), the cached timestamp stays valid. A new one is fetched only once the consumer commits a new offset.
With this approach you get:
- Accurate time lag no matter what the topic’s throughput is
- Correct handling of idle producers (time lag keeps rising, as it should)
- Limited load on Kafka thanks to caching
And the edge cases?
Two situations can skew timestamp accuracy:
Log compaction: When the topic uses cleanup.policy=compact, the message at your committed offset may have been removed by compaction. Kafka then returns the next available message, which carries a later timestamp, so the reported lag ends up slightly below the real one.
Retention deletion: When your committed offset points to a message the retention policy has already deleted, you hit the same issue - Kafka hands back a newer message.
In both cases the lag is understated (the delay shown is smaller than the real one). A sound implementation recognises these conditions and marks them, so you know the figure may be inaccurate.
TL;DR: Time lag can only be obtained by reading real messages. Interpolation looks attractive but breaks down on idle/low-throughput topics. Direct sampling combined with caching produces accurate numbers without overloading your brokers.
The metrics you really need
This is what your Kafka monitoring should cover:
Per partition - for tracking down specific problems
kafka_consumergroup_group_lag{group="orders", topic="orders", partition="0"} 1523
kafka_consumergroup_group_lag_seconds{group="orders", topic="orders", partition="0"} 12.5
Per group - for alerts
kafka_consumergroup_group_max_lag{group="orders"} 2847
kafka_consumergroup_group_max_lag_seconds{group="orders"} 45.2
Offset lag (group_lag) is still worth having – it shows how many messages are waiting to be processed. But you should combine it with time lag (group_lag_seconds) to judge how serious the situation is.
For alerts, rely on max_lag_seconds across all partitions, which gives you the group’s worst-case delay:
Fire an alert when any partition falls more than 60 seconds behind
kafka_consumergroup_group_max_lag_seconds{group="orders-processor"} > 60
This alert is triggered by real delay rather than an arbitrary message count. An SLA such as “process orders within 30 seconds” can at last be monitored the right way.
Meet klag-exporter
To tackle these problems, we built klag-exporter. It is written in Rust for performance and relies on direct timestamp sampling rather than interpolation.
How it differs from existing tools:
The exporter also recognises the compaction and retention edge cases and marks the affected partitions, so you know when the time lag should be taken with a pinch of salt.
Getting started quickly with Docker
We will begin with the simplest setup possible and expand it step by step.
Minimal setup
-
Write a basic config
-
Start the exporter
Swap your-kafka-network for the Docker network your Kafka runs on, and set bootstrap_servers to your broker(s).
- Check that it works
You should get metrics such as:
kafka_consumergroup_group_lag{cluster_name="my-cluster",group="my-group",topic="my-topic",partition="0"} 1523
kafka_consumergroup_group_lag_seconds{cluster_name="my-cluster",consumer_group="my-group",topic="my-topic",partition="0"} 12.5
Full demo stack
Would you like to see the whole thing running together - Kafka, producers, consumers, Prometheus, and Grafana - with ready-made dashboards? The repo comes with a demo you can run straight away:
The demo ships with producers and consumers that create realistic lag patterns, so you can watch the time lag and offset lag metrics at work right away.
A closer look at configuration
- Exporter settings
Granularity trade-offs:
In most setups, begin with a partition. If necessary, you can always aggregate in Prometheus.
- Cluster configuration
The whitelist/blacklist patterns are regular expressions and are applied in sequence: the whitelist first, after which the blacklist removes whatever matches.
Authentication methods
Through consumer_properties, klag-exporter supports every standard Kafka authentication mechanism.
SASL/PLAIN (username/password):
SASL/SCRAM (with TLS):
mTLS (mutual TLS, no username):
AWS MSK with IAM:
Substituting environment variables
Every config value accepts the ${VAR_NAME} syntax - so secrets never have to be hard-coded into the config file:
Deploying on Kubernetes
In production, klag-exporter will usually run in Kubernetes. Our GitHub repo contains ready-made helm charts that make deploying klag-exporter to your cluster easy.
Grafana dashboard
Importing the dashboard
Option A: Copy the JSON file
From the repository
Option B: Import through the Grafana UI
- Open Dashboards → Import
- Upload
kafka-lag.jsonor paste in its contents - Pick your Prometheus datasource
- Click Import
Main panels
The bundled dashboard is split into several sections:
Overview stats:
- Total Consumer Lag (summed over all groups)
- Max Time Lag (worst-case delay)
- Exporter Status (UP/DOWN)
- Scrape Duration (how fast collection runs)
- Consumer Groups (number of monitored groups)
Time series:
- Consumer Group Total Lag - how it changes over time
- Time Lag in Seconds - the metric that counts
- Per-Partition Lag - for tracking down specific problems
Throughput & distribution:
- Partition Latest Offsets - how they are spread across partitions
- Topic Message Rate - production throughput over time
Dashboard variables
The dashboard comes with filtering variables:
Use them to narrow things down while investigating problems.
Monitoring several clusters
A single klag-exporter instance can watch several Kafka clusters. Simply add further [[clusters]] blocks:
In the metrics, every cluster carries its own cluster_name label, which lets you filter by cluster in Grafana and in alerts.
One exporter or many – when to choose which:
OpenTelemetry integration
Besides the Prometheus endpoint (or in place of it), klag-exporter can also push metrics over OTEL.
This comes in handy if you use Datadog, New Relic, Honeycomb, or another backend that accepts OTEL ingestion.
The test stack contains an OpenTelemetry Collector that receives these metrics and exports them both to debug output (logs) and to a Prometheus endpoint.
Troubleshooting
The exporter shows no metrics
-
Check that Kafka is reachable:
-
Look at the exporter logs:
-
Confirm authentication:
Search the logs for SASL or SSL errors. Re-check your credentials and certificates.
Metrics are there, but no consumer groups
- Review your whitelist/blacklist patterns: overly strict patterns may filter groups out.
- Make sure the consumer groups exist: kafka-consumer-groups –bootstrap-server kafka:9092 –list
- Find out whether consumers have committed offsets: klag-exporter lists only groups with committed offsets.
Time lag is stuck at 0
-
Make sure timestamp sampling is turned on:
-
Find out whether consumers are lagging at all: when
group_lagis 0,group_lag_secondsis 0 too (caught up = no time lag).
Grafana displays “No data”
- Look at the Prometheus targets: http://localhost:9090/targets - is klag-exporter UP?
- Test the datasource: Grafana → Configuration → Data Sources → Test
- Check the metric names: the dashboard relies on particular metric names. If you changed them, adjust the queries.
Where you should be now
If you went through this guide, you now have:
- klag-exporter gathering both offset lag and time lag metrics
- Prometheus scraping and storing these metrics
- Grafana dashboards visualising lag trends and distributions
- Alerts set up for your SLAs
- Knowledge of the edge cases (compaction/retention detection)
Instead of “Consumer X has 50,000 messages lag,” your on-call engineers can now react to “Consumer X is 45 seconds behind” - an alert that is far easier to act on.
Main takeaways
- Offset lag shows how many messages you are behind; time lag shows how far behind you are in real time. Both are important, but your SLAs are expressed in time lag.
- Tools based on interpolation break down on idle or low-throughput topics. They report zero lag while you could be hours behind.
- Sampling timestamps directly is more accurate but requires caching. Fetch only for lagging partitions and cache the results with a TTL.
- Base alerts on time lag, not offset lag. Your on-call engineers will be grateful to grasp the severity at a glance.
If offset lag is all you monitor today, you are flying partly blind. Putting time lag on your dashboards is the single most valuable upgrade you can make to your Kafka monitoring.
The messages will stay where they are. Your users, however, are waiting - and they feel the delay.