Request Rate Increase Without Session Growth Suggests Retry Storm
warningResource ContentionUpdated Mar 19, 2025
Rising haproxy_frontend_http_requests_total without corresponding increase in haproxy_sessions indicates clients are retrying failed requests, potentially creating a feedback loop that exacerbates backend overload.
Sources
How to detect:
Calculate request-to-session ratio over time. Alert when ratio increases significantly (e.g., from 10:1 to 50:1) while haproxy_backend_retry_warnings_datadog or haproxy_server_redispatch_warnings_datadog also rise.
Recommended action:
Investigate why initial requests are failing, causing client retries. Check for backend 5xx errors, timeouts, or connection failures. Consider implementing retry budgets or circuit breakers at the application layer. Verify HAProxy retry configuration isn't overly aggressive.