Apache Kafka insights

When some brokers show significantly higher network traffic than others, it indicates uneven partition distribution or leadership imbalance, causing inefficient resource utilization.

Unbounded growth of schema subjects and versions can lead to registry performance degradation, increased memory usage, and slower schema lookups.

Frequent changes in consumer group membership trigger rebalances, causing processing pauses, increased latency, and temporary unavailability.

In KRaft mode, high Raft commit latency delays metadata changes from propagating through the cluster, causing stale metadata and operational delays.

In KRaft mode, metadata apply errors indicate the controller is failing to apply metadata changes, potentially causing inconsistent cluster state.

When active connection count approaches configured maximum, new clients will be rejected, causing connection failures and application errors.

When brokers perform message format conversion for legacy clients, it adds significant CPU overhead and latency, reducing overall throughput.

When producer requests expire before broker response, it indicates either broker overload, network issues, or producer timeout configuration is too aggressive.

When consumer fetch latency increases significantly, it indicates the broker is slow to respond to fetch requests, often due to disk I/O, CPU saturation, or competing produce traffic.

When partitions do not use their preferred leader, cluster load becomes unbalanced, reducing throughput and increasing latency as some brokers handle disproportionate leadership.

When ISR count drops below min.insync.replicas, producers configured with acks=all will fail with NotEnoughReplicasException, blocking writes to maintain durability guarantees.

When partitions go offline (all replicas unavailable), they cannot serve produce or consume requests, causing complete unavailability for affected data.

When fetch sessions are evicted frequently, it indicates consumers are not maintaining stable fetch patterns, causing the broker to rebuild fetch contexts repeatedly and increasing CPU overhead.

When ZooKeeper sessions expire, brokers re-register causing controller changes, partition leadership changes, and temporary unavailability. Frequent expirations indicate ZooKeeper or network issues.

When log flush operations take excessive time, produce requests are delayed as Kafka waits for data to be flushed to disk, impacting producer latency and throughput.

When request queue size grows, it indicates the broker cannot process incoming requests fast enough, leading to increased latency and potential client timeouts.

When producer buffer memory is exhausted, send calls block waiting for space, causing application threads to stall and reducing overall throughput.

When oldest message age approaches retention limit and consumer lag is high, messages will be deleted before consumption, causing data loss.

Lambda limits MaximumPollers to the number of Kafka topic partitions to maintain ordered processing within partitions. When a topic has few partitions relative to message volume, Lambda cannot scale event pollers sufficiently, creating a throughput ceiling regardless of provisioned capacity.

Lambda's on-demand Kafka event pollers scale based on offset lag evaluation every minute, but the autoscaling process takes up to three minutes to complete. High offset lag combined with low event poller counts indicates insufficient polling capacity before autoscaling can respond.

Kafka Consumer Lag Ingestion Backlogcritical

DataHub ingestion falling behind due to Kafka consumer lag, causing metadata changes and quality checks to be delayed, leading to stale lineage and undetected data quality issues.