Database deadlock on concurrent task_run inserts

Postgres deadlock occurs during task_run INSERT operations with ON CONFLICT clauses when multiple processes attempt concurrent inserts. Processes wait for ShareLock on transactions and speculative tokens, blocking each other. Observed in Prefect 3.3.1+ with asyncpg driver and Postgres 14.13+.

Review task_run table indexes for conflicts with ON CONFLICT (id) clause. Enable deadlock retry logic in application layer. Monitor database connection pool size via prefect.database.connections.active. Check for missing or suboptimal indexes on flow_run_id, task_key, or id columns. Consider tuning Postgres deadlock_timeout parameter. Investigate concurrent insert patterns from task_run_recorder service.