Prefect

Tasks stuck in Running state with task.submit() pattern cause flow hangs

critical
availabilityUpdated May 20, 2025(via Exa)
Technologies:
How to detect:

Tasks submitted using task.submit() pattern become stuck in 'Running' state indefinitely and never complete, causing flows to hang when calling future.result(). Issue is non-deterministic - sometimes 1-3 tasks out of 10 get stuck with no errors or logs. Tasks appear as Running in Prefect UI but never transition to completed. Affects ConcurrentTaskRunner, ThreadPoolTaskRunner, and RayTaskRunner. Occurs frequently with high concurrency flow runs in Prefect 3.4.x.

Recommended action:

This is an open bug (GitHub #18121) with no known workaround as of Prefect 3.4.12.dev2. Monitor prefect.task_run.duration and prefect.flow_run.duration for abnormally long-running tasks. Implement timeouts on future.result() calls to prevent indefinite hangs. Consider staying on or reverting to 3.1.12 if issue blocks production. Reducing batch sizes, lowering max_workers, adding delays between submissions, and switching task runners did not resolve the issue. Track the GitHub issue for updates from maintainers.