Elasticsearch insights

Slow shard recovery after node failures or restarts delays cluster stabilization and can indicate network issues, disk I/O bottlenecks, or configuration problems.

Long-running or stuck tasks accumulate in the task manager, potentially blocking critical operations and consuming resources.

Excessive script compilation from unique painless scripts can exhaust the compilation cache, trigger rate limiting, and impact query performance.

Improper distribution of shards or unbalanced node roles can cause resource hotspots where some nodes are overloaded while others are underutilized.

High latency for document GET operations indicates potential issues with routing, shard location, or storage performance when retrieving specific documents by ID.

Complex ingest pipelines with processors like grok, script, or ML inference can significantly slow document indexing, causing backpressure and increased latency.

Cross-cluster search queries can cause significant coordination overhead, network traffic, and timeout issues when remote clusters are slow or have high latency.

ILM policies failing to execute transitions (hot to warm, warm to cold, delete) causes indices to accumulate in wrong tiers, wasting resources and degrading performance.

Failed snapshot operations risk data loss and violate backup SLAs. SLM policy failures can result from repository unavailability, insufficient permissions, or storage quota issues.

High network throughput or connection count on cluster transport layer indicates heavy inter-node communication that can saturate network bandwidth and cause coordination delays.

Too many shards consume cluster resources even when idle, causing slow queries, increased overhead, and reduced stability. Rule of thumb: keep shards below 20 per GB of heap configured.

Transaction log (translog) accumulates uncommitted operations between flushes. Excessive translog size increases recovery time after node failures and can indicate flush problems or configuration issues.

Slow cluster state updates cause delays in shard allocation, index creation, and mapping updates. Large cluster state size or master node resource constraints are common causes.

Search operations taking longer than acceptable thresholds (typically >500ms for p95) indicate performance issues from oversized result sets, complex aggregations, inefficient queries, or resource contention.

Background segment merges consolidate small Lucene segments into larger ones, reducing file count. When merge rate cannot keep pace with segment creation, segment count explodes causing slow queries and increased memory usage.

Circuit breakers trip to prevent operations that would cause OutOfMemoryError by estimating memory requirements and rejecting requests that exceed configured limits. Frequent trips indicate memory pressure or oversized operations.

Heavy bulk indexing workload causes high CPU utilization, memory pressure, and increased indexing latency, potentially impacting concurrent search performance and cluster stability.

Elasticsearch Indexing Lag Metadata Stalecritical

DataHub metadata search results and lineage views showing stale information because Elasticsearch indices are not being updated timely, impacting data discovery and incident response.

Trace Completion Rate Degradationcritical

When the ratio of saved traces to received spans drops below 95%, traces are being lost before storage, indicating storage backend issues, network timeouts, or collector resource constraints.

Storage Backend Write Latency Bottleneckcritical

Slow storage write operations block collector workers, causing span reception to slow and queues to back up, ultimately leading to dropped traces.

Query Service Latency Degradationwarning

High P95/P99 query latencies (>5 seconds) make Jaeger UI unusable during incident troubleshooting, typically caused by slow storage reads, overloaded shards, or inefficient trace queries.

When JVM heap usage stays above 85% for extended periods, garbage collection pauses increase dramatically, leading to node unresponsiveness, cluster state propagation failures, and potential split-brain scenarios.

When any node crosses the low disk watermark (85% full by default), Elasticsearch starts relocating shards. Multiple nodes hitting watermarks simultaneously can trigger cascading relocations that overload cluster I/O and delay recovery.

Requests for high-offset results (e.g., page 500 with OFFSET 5000) force every shard to fetch and score all preceding documents before discarding them, causing coordinating node memory pressure and exponential latency growth beyond ~10K results.

Expensive queries or indexing operations monopolize thread pool workers, causing benign requests to queue indefinitely. Manifests as stuck tasks in _cat/tasks with millisecond operations taking minutes while thread pools show 100% utilization.

When primary shards cannot be assigned (elasticsearch.cluster.health == 2), data becomes unavailable and cluster enters red state. This occurs from insufficient nodes, misconfigured shard allocation rules, or node failures during insufficient replica coverage.

Refresh operations make indexed documents searchable by writing in-memory buffer to disk segments. Default 1-second interval creates overhead that scales with indexing rate. When refresh time exceeds interval, indexing throughput collapses and latency spikes.

Background segment merges consolidate small Lucene segments into larger ones, reducing file count but consuming I/O. Default throttling (20MB/s) prevents merge backlog from overwhelming cluster, but excessive throttling causes segment explosion and query performance degradation.

Field data (inverted reverse index for aggregations) loads into JVM heap on first access and persists for segment lifetime. When circuit breaker limit or cache size is too small, frequent evictions cause repeated expensive field data loading, spiking CPU and heap pressure.