Topic Retention Approaching With Insufficient Consumer Throughput
criticalWhen oldest message age approaches retention limit and consumer lag is high, messages will be deleted before consumption, causing data loss.
Calculate oldest unconsumed message age using kafka.partition.oldest_offset, kafka.consumer_group.offset, and message timestamps. Alert when age > 0.8 * kafka.topic.config.retention_ms and lag is increasing.
1. Immediate: Increase kafka.topic.config.retention_ms to buy time. 2. Scale consumers: Add consumer instances to increase throughput. 3. Optimize processing: Reduce per-message processing time. 4. Review partition count: Increase partitions to enable more parallelism. 5. Check for stuck consumers: Identify and restart any stalled consumer instances. 6. Consider retention policy: Evaluate if retention is appropriate for use case.