Kafka Partition Imbalance in Lambda Event Processing

critical

scalingUpdated Feb 23, 2026

Lambda limits MaximumPollers to the number of Kafka topic partitions to maintain ordered processing within partitions. When a topic has few partitions relative to message volume, Lambda cannot scale event pollers sufficiently, creating a throughput ceiling regardless of provisioned capacity.

Sources

Apache Kafka event poller scaling modes in Lambda - AWS Lambdadocs.aws.amazon.com

Technologies:

Apache KafkaThe root cause of this issue originates in Apache Kafka

kafka.topic.partitions

kafka.topic.messages_in.rate

kafka.consumer.max_lag

How to detect:

Monitor when Kafka topic partition count is low (< 10) while message rate and offset lag are high. Check if Lambda ProvisionedPollers or event poller metrics are capped at a value equal to partition count despite MaximumPollers being configured higher.

Recommended action:

Increase Kafka topic partition count to enable Lambda horizontal scaling. Calculate required partitions by dividing expected throughput (MB/sec) by per-poller capacity (5 MB/sec). After repartitioning, adjust Lambda MaximumPollers to match new partition count. Monitor ProvisionedPollers to confirm scaling now occurs.