Compaction Failures Prevent Log Cleanup and Increase Storage Costs
warningElevated pulsar_compaction_failed combined with low pulsar_compaction_write_throughput indicates topic compaction is failing or not running, preventing cleanup of superseded messages and causing unbounded storage growth for compacted topics.
Monitor pulsar_compaction_failed for repeated failures. Check pulsar_compaction_write_throughput to confirm compaction is running. Correlate with pulsar_storage_logical_size growth on compacted topics to assess storage impact. Review pulsar_compaction_time_time_in_mills for compaction duration issues.
Review compaction scheduler configuration and ensure automatic compaction is enabled with appropriate frequency. Check compaction logs for specific error messages. Verify sufficient broker resources (CPU, memory) for compaction tasks. Consider manually triggering compaction via pulsar-admin for critical topics. Monitor storage growth rate and adjust data retention policies if compaction cannot keep pace.