Tag concurrency limit deadlock with dependent tasks in Prefect 3.2.15+
criticalResource ContentionUpdated Apr 1, 2025(via Exa)
Technologies:
How to detect:
In Prefect versions 3.2.15 through 3.3.1, submitted tasks using wait_for dependencies immediately consume concurrency slots upon submission rather than at execution start. When concurrency slots are exhausted by pending tasks waiting for dependencies, the tasks they depend on cannot acquire slots to execute, creating complete deadlock. Specific to ConcurrentTaskRunner.
Recommended action:
Downgrade to Prefect 3.2.14 or earlier, or upgrade to version including fix from PR #17943. Workarounds: switch to DaskTaskRunner via @flow(task_runner=DaskTaskRunner()), or convert to fully asynchronous tasks and flows with async/await pattern.