PrefectPostgreSQL

PostgreSQL connection pool exhaustion causes Prefect server errors and worker crashes

critical
Connection ManagementUpdated Mar 3, 2025(via Exa)
How to detect:

Prefect server exhausts PostgreSQL connection slots when multiple workers and background services connect, triggering asyncpg.exceptions.TooManyConnectionsError. This causes 500 Internal Server Errors from the API and cascading worker crash loops (690+ restarts observed). Idle workers increase connection demand even without active flow runs. Connection poolers like pgbouncer may exacerbate the issue if misconfigured with insufficient connection limits.

Recommended action:

1) Check PostgreSQL max_connections setting and query current active connections. 2) Tune Prefect's pool_size and max_overflow settings to limit per-process connection usage. 3) If using pgbouncer or similar connection pooler, verify pool configuration can accommodate Prefect server + all workers. 4) As immediate mitigation, restart prefect-server and postgresql to clear stale connections. 5) Monitor prefect.database.connections.active metric to detect approaching connection limits before exhaustion.