Amazon S3 insights

Terraform-Managed Lambda Concurrency Exhaustion Leading to S3 Throttling Cascade

When Lambda functions managed by Terraform reach reserved concurrency limits while processing S3 events, backpressure causes S3 request metrics to spike and queue depth to grow in dependent SQS queues, creating a cascading failure pattern across infrastructure components.

High FirstByteLatency indicates S3 is taking excessive time to begin responding to requests, often caused by request rate throttling (503 SlowDown errors) or sequential key patterns creating partition hotspots.

Sudden increases in 5xx errors (especially 503 SlowDown) indicate the application has exceeded S3's per-prefix request rate limits of 3,500 PUT/POST/DELETE or 5,500 GET/HEAD requests per second.

When TotalRequestLatency significantly exceeds FirstByteLatency, it indicates network transfer bottlenecks or client-side processing delays rather than S3 service issues.

Elevated 4xx client error rates typically indicate IAM policy misconfigurations, bucket policy issues, or ACL problems preventing legitimate access. These errors can cascade across dependent services and indicate security or integration problems.

High FirstByteLatency combined with 503 SlowDown errors indicates S3 request throttling due to exceeding partition-level request rate limits. This commonly occurs with sequential key patterns or batch processing jobs making thousands of small random-access requests.

Exponential growth in BucketSizeBytes for Standard storage class despite stable user growth indicates inefficient lifecycle policies or logging systems storing data indefinitely. This leads to unnecessary storage costs without business value.

Upload timeouts and failures for files larger than 100MB indicate missing multipart upload implementation. Standard PutObject has a 5GB limit and lacks resilience for network interruptions, causing complete upload restarts on any failure.

Sustained growth in BucketSizeBytes despite lifecycle policies suggests misconfigured transitions, expired objects not being deleted, or versioning consuming storage without cleanup.

S3 Lifecycle Policy Drift Causing Unexpected Storage Cost Increases

When S3 BucketSizeBytes grows exponentially for Standard storage class despite stable workload patterns, it often indicates S3 Lifecycle policies managed by Terraform are not being applied correctly, missing, or have drifted from desired state.