Prefect

CancellationCleanup service execution exceeds loop interval

warning
performanceUpdated Apr 8, 2025(via Exa)
Technologies:
How to detect:

CancellationCleanup service consistently takes 50-70 seconds to execute when configured loop interval is 20 seconds. Execution time is 2.5-3.5x the expected interval, causing service backlog and delayed cleanup of cancelled flow/task runs.

Recommended action:

Query Postgres slow query logs for CancellationCleanup operations. Check for growth in pending cancellations requiring cleanup. Review indexes on cancellation-related state tables. Consider increasing loop interval if cleanup volume has permanently grown. Monitor prefect.database.query.duration for cleanup queries.