Hot Partition Creates Uneven Consumer Lag
warningWhen one partition consistently shows higher lag than others, it indicates uneven key distribution or specific message types requiring more processing time, creating a processing bottleneck.
Compare kafka.consumer.max_lag across partitions for the same consumer group. Alert when one partition's lag exceeds others by 2x or more consistently. Use partition-level offset metrics to identify the specific hot partition.
Review partition key distribution to ensure even spread across partitions. If specific keys cause heavy processing, consider repartitioning with different key logic. For Lambda ESM, ensure sufficient provisioned pollers or increase maximum pollers to handle uneven partition load. Profile message processing to optimize slow message types.