PostgreSQL connection pool exhaustion causes Prefect server errors and worker crashes
criticalPrefect server exhausts PostgreSQL connection slots when multiple workers and background services connect, triggering asyncpg.exceptions.TooManyConnectionsError. This causes 500 Internal Server Errors from the API and cascading worker crash loops (690+ restarts observed). Idle workers increase connection demand even without active flow runs. Connection poolers like pgbouncer may exacerbate the issue if misconfigured with insufficient connection limits.
1) Check PostgreSQL max_connections setting and query current active connections. 2) Tune Prefect's pool_size and max_overflow settings to limit per-process connection usage. 3) If using pgbouncer or similar connection pooler, verify pool configuration can accommodate Prefect server + all workers. 4) As immediate mitigation, restart prefect-server and postgresql to clear stale connections. 5) Monitor prefect.database.connections.active metric to detect approaching connection limits before exhaustion.