Disk Watermark Shard Relocation Storm
warningWhen any node crosses the low disk watermark (85% full by default), Elasticsearch starts relocating shards. Multiple nodes hitting watermarks simultaneously can trigger cascading relocations that overload cluster I/O and delay recovery.
elasticsearch.node.fs.disk.available / elasticsearch.node.fs.disk.total ratio dropping below 0.15 (85% full) on any node, combined with rising elasticsearch.cluster.shards in 'relocating' state
Set up proactive alerts at 80% disk usage with 15%+ free space buffer. When triggered: (1) Add nodes or increase disk capacity before hitting high watermark (90%), (2) Review and delete old indices or implement ILM policies (elasticsearch.ilm metrics), (3) Temporarily increase cluster.routing.allocation.disk.watermark settings if in emergency (check elasticsearch.cluster.settings). Use _cat/allocation API to identify which nodes are full. For time-series data, implement hot/warm/cold architecture to move old data to cheaper storage. Monitor elasticsearch.node.shards.size to track shard distribution.