kafka.broker.config.log_retention_bytes

The kafka.broker.config.log_retention_bytes metric reports the configured maximum size in bytes that a log partition can grow to before Kafka begins deleting old log segments to free up space. This is a broker-level configuration parameter that directly controls disk space utilization by setting a size-based retention policy for topic partitions. When a partition's total log size exceeds this threshold, Kafka's log retention mechanism will remove the oldest segments first, regardless of their age. This metric is exposed as a gauge representing the current configuration value on each broker, and it plays a critical role in capacity planning and preventing disk exhaustion scenarios. The configuration corresponds to the log.retention.bytes broker setting, which can be overridden at the topic level using the retention.bytes topic configuration.

From an operational perspective, monitoring this metric is essential for understanding storage allocation policies and ensuring alignment between configured retention limits and actual disk capacity. Operators should correlate this value with actual partition sizes and available disk space to prevent out-of-disk errors that would render brokers inoperable. A value of -1 (the default) indicates no size-based retention limit, meaning retention is controlled exclusively by time-based settings like log.retention.hours. In production environments, setting appropriate retention byte limits helps with cost management by preventing unbounded storage growth, particularly for high-throughput topics where time-based retention alone might allow partitions to consume excessive disk space before cleanup occurs.

Common alerting and troubleshooting scenarios involve comparing this configuration metric against actual partition sizes (monitored via metrics like kafka.log.log_size) and available disk capacity. If partition sizes consistently approach or exceed the configured retention bytes, it may indicate that the retention policy is too aggressive, potentially causing premature data loss, or that producers are generating data faster than expected. Conversely, if retention bytes are set extremely high relative to available storage, operators risk disk exhaustion before retention policies can take effect. Organizations should establish alerting when actual log sizes reach 80-90% of the configured log_retention_bytes value or when the combined retention limits across all partitions on a broker exceed available disk capacity with appropriate safety margins. The Apache Kafka Operations documentation provides additional guidance on capacity planning and retention policy configuration for production deployments.