Compaction & Retention: Corner Cases That Distort Your Kafka Lag Metrics
Can you trust your Kafka lag dashboard? Find out how log compaction and retention can mask huge consumer delays and how to spot misleading time lag metrics. Teams frequently rely on Kafka lag metrics to track system health, yet log compaction and retention policies can quietly remove messages before anyone processes them. This article explains why the usual time lag formula – $current\_time - timestamp\_of\_message\_at\_committed\_offset$ – underestimates actual latency once the requested offset is gone. We describe how "klag-exporter" spots these inaccuracies by comparing requested and actual offsets as well as low watermarks. Lastly, we offer practical mitigation strategies, such as tuning min.compaction.lag.ms, optimizing consumer resources, and adding warning indicators to dashboards for more resilient Kafka monitoring.
According to your dashboard, the inventory-updates consumer is 5 seconds behind. All seems well. Yet users keep reporting stale data – the inventory counts they see are 10 minutes old.
You go back over the metrics. Still 5 seconds. By the numbers, you have almost caught up.
The numbers are not telling the truth.
Time lag is more important than offset lag, but there is a caveat: even a correctly calculated time lag can deceive you. Two Kafka features – log compaction and retention – are capable of deleting the exact messages your metrics rely on, which makes the reported lag much lower than the real one.
Take a look at our new tool - a high-performance Apache Kafka consumer group lag exporter written in Rust. It computes both offset lag and time lag (latency in seconds) using precise timestamp-based measurements.
Let me walk you through how this happens and what you can do to address it.
Quick Recap: How Time Lag Works
On paper, calculating time lag is simple:
time_lag = current_time - timestamp_of_message_at_committed_offset
To obtain that timestamp, we seek to the consumer’s committed offset and read the message stored there. Easy.
But what happens when that message is no longer there?
Kafka does not raise an error. Instead, it quietly hands back the next message that is available. That message carries a later timestamp, so the lag we calculate appears smaller than it really is.
Imagine judging how far a runner is behind by looking at the last checkpoint marker they passed. If someone wiped out markers 1 through 10 and the first one you can see is 11, you would log the runner at checkpoint 11 – even if they are actually still at checkpoint 1.
That is precisely what compaction and retention do.
Log Compaction Problem
Log compaction is a Kafka feature used by topics set up with cleanup.policy=compact. Rather than storing every message, Kafka keeps only the most recent message per key and removes the older versions.
It is ideal for use cases such as:
- Change Data Capture (CDC) streams
- Topics backing KTables
- Configuration or state stores
This is how compaction alters your partition:
Before compaction:
offset 100: key=user-1, value={"status":"active"}, ts=10:00:00
offset 101: key=user-2, value={"status":"pending"}, ts=10:00:15
offset 102: key=user-1, value={"status":"inactive"}, ts=10:00:30 ← newer user-1
offset 103: key=user-3, value={"status":"active"}, ts=10:00:45
After compaction:
offset 101: key=user-2, value={"status":"pending"}, ts=10:00:15
offset 102: key=user-1, value={"status":"inactive"}, ts=10:00:30
offset 103: key=user-3, value={"status":"active"}, ts=10:00:45
(offset 100 is gone - compacted away)
The Lag Problem
Suppose your consumer committed offset 100 before compaction kicked in. You want to compute the time lag:
- Seek to offset 100
- Kafka cannot locate offset 100 (compaction removed it)
- Kafka hands back offset 101 in its place
- You read the timestamp 10:00:15
- With the current time at 10:01:00, you report 45 seconds of lag
In reality, though, the consumer sits at offset 100, produced at 10:00:00. The true lag is 60 seconds – 25% more than what was reported.
In this basic example, the error is minor. In production, it can be enormous.
Real-World Impact
Consumers of compacted topics are frequently slow. A textbook case: rebuilding a KTable from scratch. Your consumer may be sitting at offset 0, slowly working through millions of messages while compaction keeps removing old data behind the latest offset.
A possible scenario:
- Reported time lag: 30 seconds
- Actual time lag: 30 minutes
The consumer was rebuilding its state from the beginning on a heavily compacted topic. Thousands of early messages had already been compacted away, so each timestamp lookup came back with a message far newer than the consumer’s real position.
Retention Deletion: When Your Offsets Fall Off the Cliff
Whereas compaction removes particular messages (older values for the same key), retention deletes messages from the start of the log according to time or size limits.
The default policy (cleanup.policy=delete) deletes messages once retention.ms has passed or when retention.bytes is exceeded. The oldest messages are always deleted first.
Day 1 (with 7-day retention configured):
low_watermark = 0
offset 0: ts=Day1 09:00
offset 1: ts=Day1 09:01
offset 2: ts=Day2 10:00
...
high_watermark = 1000
Day 8:
low_watermark = 500 ← moved forward!
offset 500: ts=Day2 12:00 ← now the earliest message
...
high_watermark = 5000
(offsets 0-499 deleted by retention)
The Lag Problem
Suppose your consumer committed offset 50 before those messages got deleted:
- The consumer’s committed offset: 50
- Kafka’s low watermark: 500 (offsets 0-499 no longer exist)
- You attempt to seek to offset 50
- Kafka hands back offset 500 in its place
- The timestamp you read comes from Day 2, not Day 1
- The reported lag is wrong by a day or more
Why This Happens
Typical situations that cause retention-based inaccuracy:
- Consumer halted for days or weeks – Someone switched it off during a migration and never switched it back on.
- Slow consumer on a high-volume topic – Processing falls behind, and old messages expire before they get consumed.
- Consumer offset reset to 0 – A deliberate replay, but the older messages have already been deleted.
In each of these cases, your time lag metric will severely underestimate how far behind the consumer actually is.
Detection: How klag-exporter Catches These Lies
Fortunately, both situations can be detected. All you have to do is compare what you requested with what Kafka really returned.
Compaction Detection
requested_offset = committed_offset
actual_offset = offset_of_returned_message
if actual_offset > requested_offset:
# Gap detected - compaction removed messages
compaction_likely = true
If we seek to offset 100 but receive offset 101, we can tell that something has been compacted away.
Retention Detection
This case is simpler still – the low watermark (the earliest available offset) is already fetched during normal operation:
if committed_offset < low_watermark:
# Consumer's position is before the earliest message
retention_affected = true
When your committed offset is 50 and the low watermark is 500, those messages simply no longer exist.
What klag-exporter Exposes
klag-exporter keeps track of both situations automatically:
Labels on lag metrics, per partition:
kafka_consumergroup_group_lag_seconds{..., compaction_detected="true"} 45.2
kafka_consumergroup_group_lag_seconds{..., retention_detected="true"} 120.5
Detection counters:
kafka_lag_exporter_compaction_detected_total 12
kafka_lag_exporter_retention_detected_total 3
Each time one of these situations is detected during a scrape, the matching counter goes up. If they are above zero, you know that some of your time lag figures cannot be trusted.
What to Do About It
Let’s look at the solutions available to us.
Mitigation for Compacted Topics
The simplest and most direct option is to raise min.compaction.lag.ms, which naturally postpones compaction and gives your consumers more time to catch up before messages vanish. The default is 0 (eligible immediately).
Another option is to make sure consumers keep pace - if your consumer routinely lags behind compaction, fix the consumer (scale it, optimize it, or allocate more resources to it).
Finally, you can simply live with the limitation – for compacted topics, read time lag as “at least this much” rather than as a precise measurement.
Mitigation for Retention-Affected Consumers
When a consumer’s offset is behind the low watermark, inaccurate metrics are the least of your worries.
The biggest issue is that the consumer has missed data, and those messages are lost for good. You have to decide whether that is acceptable or whether you need to replay from a different source. You can of course carry on from whatever is still available, but more importantly, dig into the root cause – why the consumer fell so far behind (for example, it was stopped or too slow) – and then fix that underlying problem.
For Monitoring Dashboards
Whenever detection flags are set:
- Present offset lag as the main metric (it remains accurate)
- Show time lag together with a warning indicator
- Make it clear that the value is a lower bound
Log compaction and retention can remove messages that your consumer has not processed yet. When that happens, time lag metrics underestimate the problem – sometimes drastically. klag-exporter detects both situations and flags them, so you know when to dig deeper instead of blindly relying on the numbers.
Now that you know both what time lag offers and where it falls short, you are ready to deploy klag-exporter in your own environment.