ProcessPoolExecutor processes hold memory indefinitely without max_tasks_per_child
infoResource ContentionUpdated May 21, 2023(via Exa)
How to detect:
When Flask app uses Python's ProcessPoolExecutor within Gunicorn workers, child processes may accumulate memory over time without being restarted, leading to increased memory usage and degraded performance across successive load tests.
Recommended action:
Set max_tasks_per_child parameter on ProcessPoolExecutor to 500. This causes processes to exit and be replaced with fresh workers after executing 500 tasks, releasing any held memory. Per Python docs: 'max_tasks_per_child specifies the maximum number of tasks a single process can execute before it will exit and be replaced with a fresh worker process.'