Elasticsearch Indexing Lag Metadata Stale
criticalDataHub metadata search results and lineage views showing stale information because Elasticsearch indices are not being updated timely, impacting data discovery and incident response.
Monitor elasticsearch_index write latency and throughput. Detect when metadata updates (from Kafka consumers) are not reflected in search indices within acceptable time windows (e.g., >5 minutes delay). Check if datasetindex_v2, corpuserindex_v2, and other DataHub indices show update lag compared to Kafka topic timestamps.
Investigate Elasticsearch cluster health - check for resource saturation (CPU, memory, disk), indexing queue depth, and bulk indexing failures. Review DataHub MAE consumer logs for Elasticsearch connection errors or timeout exceptions. Scale Elasticsearch cluster or optimize index settings (refresh interval, replica count). Monitor metadata_change_proposal_process_time to identify if DataHub processing is bottleneck vs Elasticsearch write performance.