Kafka Consumer Lag Ingestion Backlog

critical

reliabilityUpdated Feb 23, 2026

DataHub ingestion falling behind due to Kafka consumer lag, causing metadata changes and quality checks to be delayed, leading to stale lineage and undetected data quality issues.

Sources

Monitoring DataHubdocs.datahub.com

Quickstart Debugging Guide | DataHubdatahubproject.io

Technologies:

DataHubSymptoms of this issue are visible in DataHub metrics and logs

Apache KafkaThe root cause of this issue originates in Apache Kafka

How to detect:

Monitor kafka_consumer_lag for DataHub's MCE (MetadataChangeEvent), MAE (MetadataAuditEvent), MCP (MetadataChangeProposal_v1), and MCL (MetadataChangeLog_v1) consumers. Alert when lag exceeds thresholds (e.g., >1000 messages for 5+ minutes) indicating metadata updates are not being processed in real-time.

Recommended action:

Scale DataHub MCE/MAE consumer pods horizontally or increase consumer throughput. Check messaging_process_time to identify if processing is slow due to downstream bottlenecks (Elasticsearch indexing, database writes). Verify Kafka broker health and topic partition distribution. Review recent metadata volume spikes using kafka_message_queue_time to determine if backlog is temporary or sustained.