DataHub insights

DataHub backend API experiencing elevated error rates impacting metadata ingestion, UI operations, and external integrations, potentially indicating service degradation or infrastructure issues.

DataHub GMS or Frontend experiencing memory pressure causing frequent garbage collection pauses, degrading API response times and potentially leading to OutOfMemoryErrors and service unavailability.

DataHub ingestion jobs failing silently or with warnings, causing metadata gaps that prevent data quality monitoring, lineage tracking, and incident detection from functioning properly.

Elasticsearch Indexing Lag Metadata Stalecritical

DataHub metadata search results and lineage views showing stale information because Elasticsearch indices are not being updated timely, impacting data discovery and incident response.

Kafka Consumer Lag Ingestion Backlogcritical

DataHub ingestion falling behind due to Kafka consumer lag, causing metadata changes and quality checks to be delayed, leading to stale lineage and undetected data quality issues.

Slow GraphQL queries hiding which specific nested fields (lineage, schemaMetadata, owner) are causing latency, making optimization impossible without path-level instrumentation.

DataHub performance degrades immediately after code deployments due to introduced regressions, configuration changes, or schema migrations. Traditional metrics show symptoms but don't correlate with deployment timing.

Organizations experiencing data quality incidents (bad data reaching dashboards, ML models) that DataHub's observability should catch, but failures occur because assertions are not configured or monitored on critical datasets.

When upstream data quality issues occur, teams cannot quickly identify downstream impact (which dashboards, ML models, reports are affected) because column-level lineage is incomplete or not queried during incident response.

Elasticsearch Memory Pressure from Unindexed Schema Fieldscritical

DataHub's search performance degrades when Elasticsearch cluster memory is exhausted by unoptimized index mappings or excessive schema fields. High heap usage combined with slow search queries indicates index buffer or field data cache exhaustion.

JVM GC Pause Cascading to Kafka Consumer Lagwarning

Prolonged JVM garbage collection pauses in DataHub consumers cause Kafka consumer lag to spike. This creates a cascading failure where metadata ingestion stalls, leading to stale catalog data and failed data quality checks.

DataHub experiences severe latency spikes immediately after pod restarts when entity cache is cold. Every GraphQL query hits the database directly, causing connection pool exhaustion and cascading timeouts.

Ingestion Source Backpressure from Sink Write Throttlingwarning

DataHub ingestion pipelines stall when sink write rate (sink_workunits_write) lags significantly behind source production rate (source_workunits_produced). This indicates downstream storage systems (Elasticsearch, database) cannot keep pace with ingestion volume.

Kafka Consumer Lag Masking Ingestion Failurescritical

DataHub's asynchronous write architecture can hide processing failures. High Kafka consumer lag combined with ingestion warnings/failures indicates metadata events are queued but not successfully persisting to primary or search storage.