Datadog insights

Storage I/O Queue Saturation Cascade

Disk queue depth (iopsinprogress) above 5 sustained indicates storage cannot keep up with demand. When combined with low actual IOPS/MBPS, points to provisioning issues or CPU bottleneck preventing I/O generation.

CockroachDB Range Under-Replication Risk

Under-replicated ranges indicate data not conforming to resilience goals. Combined with critical node detection, this reveals single points of failure that could cause data unavailability.

ECS Task Placement Failure from Cluster Resource Exhaustion

High CPU and memory reservation metrics indicate cluster capacity nearly allocated. Tasks remain pending, services cannot scale, latency climbs. Requires distinguishing whether actual utilization justifies reservation or resources are over-reserved relative to usage.

Fargate Task Definition Drift Causing Post-Deployment Regression

Behavior changes after deployment require comparing task definition versions and container images. Side-by-side diff in ECS Explorer highlights configuration changes or image tag updates that introduced issues, enabling rapid rollback decisions.

Fargate Ephemeral Storage Exhaustion Causing Task Failures

Fargate tasks use ephemeral storage for logs, temporary files, and caches. When space runs out, tasks fail. Default monitor detects this condition, requiring task definition adjustment of ephemeralStorage.sizeInGiB value.

ECS Fargate OOM Kill Despite Reported Memory Headroom

Containers killed with exit code 137 (OOM) even when Datadog reports usage well below limits. Discrepancy between Datadog-reported metrics and actual container runtime memory accounting reveals hidden memory pressure from JVM GC, fragmentation, or unmeasured kernel overhead.

Log Stream Delivery Failure Data Gap

Failed log delivery to external analytics platforms (Datadog, Splunk, etc.) creates blind spots in security monitoring, compliance auditing, and performance analysis. Failures may go unnoticed until needed for incident investigation.

Fargate Task Overprovisioning Through Underutilization Detection

When CPU and memory utilization consistently remain far below provisioned resources, tasks are overprovisioned. This drives unnecessary costs since Fargate pricing is based on allocated (not used) resources. Rightsizing requires correlating utilization with request/limit specifications.