kafka_consumer_lag

DataHub's asynchronous write architecture can hide processing failures. High Kafka consumer lag combined with ingestion warnings/failures indicates metadata events are queued but not successfully persisting to primary or search storage.

Prolonged JVM garbage collection pauses in DataHub consumers cause Kafka consumer lag to spike. This creates a cascading failure where metadata ingestion stalls, leading to stale catalog data and failed data quality checks.

DataHub ingestion falling behind due to Kafka consumer lag, causing metadata changes and quality checks to be delayed, leading to stale lineage and undetected data quality issues.