Ingest Pipeline Processing Bottleneck
warningComplex ingest pipelines with processors like grok, script, or ML inference can significantly slow document indexing, causing backpressure and increased latency.
elasticsearch.node.ingest.operations.failed increasing or elasticsearch.node.ingest.documents.current staying elevated with high elasticsearch.node.pipeline.ingest.documents.preprocessed processing time
Profile ingest pipelines via _ingest/pipeline/_simulate with verbose:true. Common issues: (1) Grok patterns too complex - simplify or use simpler parsers, (2) Script processor CPU-intensive - optimize painless scripts or move logic to application, (3) Enrich processor cache misses - increase enrich.cache_size, (4) ML inference slow - increase model allocations and threads (for ELSER etc). Monitor elasticsearch.node.ingest.documents per pipeline to identify bottlenecks. Consider splitting pipelines into smaller stages or processing documents before indexing. For high failure rates, check document format matches pipeline expectations. Use dedicated ingest nodes to isolate processing from data nodes.