Amazon ECS insights

Invalid ECS Task CPU/Memory Configuration Blocking Terraform Deployments

Terraform apply failures or ECS task launch failures occur when task definitions specify CPU and memory combinations not supported by Fargate, causing deployment rollbacks and service unavailability.

Tasks experiencing high CPU utilization (>70-80%) lead to thread starvation, execution timeouts, and clock leap issues. Sustained CPU pressure causes request latency increases and degraded application responsiveness.

Service fails to scale out fast enough during traffic surges due to conservative scaling policies or cooldown periods. Tasks hit resource limits before auto scaling provisions additional capacity, leading to degraded performance.

Terraform-Provisioned Consumer Lag Detected via SQS Queue Depth and Message Age

SQS queue depth (ApproximateNumberOfMessages) growing alongside ApproximateAgeOfOldestMessage indicates consumers provisioned via Terraform infrastructure-as-code are failing to keep pace with producers, often due to misconfigured CPU/memory limits in ECS task definitions or Lambda concurrency caps.

ECS Task Placement Failure from Cluster Resource Exhaustion

High CPU and memory reservation metrics indicate cluster capacity nearly allocated. Tasks remain pending, services cannot scale, latency climbs. Requires distinguishing whether actual utilization justifies reservation or resources are over-reserved relative to usage.

Fargate Task Definition Drift Causing Post-Deployment Regression

Behavior changes after deployment require comparing task definition versions and container images. Side-by-side diff in ECS Explorer highlights configuration changes or image tag updates that introduced issues, enabling rapid rollback decisions.

Fargate Ephemeral Storage Exhaustion Causing Task Failures

Fargate tasks use ephemeral storage for logs, temporary files, and caches. When space runs out, tasks fail. Default monitor detects this condition, requiring task definition adjustment of ephemeralStorage.sizeInGiB value.

ECS tasks consuming memory beyond allocated limits trigger out-of-memory kills, discarding in-flight operations and causing data loss. High memory utilization (>85%) correlates with task instability and sudden stops.

Tasks stuck in PENDING state when container instances lack sufficient CPU or memory capacity. Service deployments fail with RESOURCE:* errors when available cluster resources cannot satisfy task requirements.

Containers cannot reach external services or AWS endpoints, resulting in connection timeouts, DNS resolution failures, and SocketTimeoutException errors. Security group misconfigurations or missing NAT gateway routes are common causes.

Container instances show as disconnected in ECS console when the ECS agent cannot communicate with service endpoints. Prevents task placement, health reporting, and deployment operations on affected instances.

Tasks on Fargate exhaust ephemeral storage limits, causing write failures and container crashes. Storage metrics show ecs_fargate_ephemeral_storage_utilized approaching ecs_fargate_ephemeral_storage_reserved, while I/O operations spike.

ECS Fargate OOM Kill Despite Reported Memory Headroom

Containers killed with exit code 137 (OOM) even when Datadog reports usage well below limits. Discrepancy between Datadog-reported metrics and actual container runtime memory accounting reveals hidden memory pressure from JVM GC, fragmentation, or unmeasured kernel overhead.

In multi-container tasks, individual containers may exhaust resources while task-level metrics appear healthy. Requires container-level granularity to identify which container is causing resource contention.

Application load balancer health checks fail due to insufficient HealthCheckGracePeriodSeconds or incorrect HealthCheckPath configuration. Tasks are repeatedly killed and restarted, preventing deployment from reaching steady state.

Tasks fail to start with CannotPullContainerError when ECS cannot retrieve container images from ECR or Docker Hub. Common causes include IAM permission issues, network connectivity problems, or incorrect image URIs.

Fargate Task Overprovisioning Through Underutilization Detection

When CPU and memory utilization consistently remain far below provisioned resources, tasks are overprovisioned. This drives unnecessary costs since Fargate pricing is based on allocated (not used) resources. Rightsizing requires correlating utilization with request/limit specifications.