Prefect

MarkLateRuns service execution exceeds loop interval

warning
performanceUpdated Apr 8, 2025(via Exa)
Technologies:
How to detect:

MarkLateRuns service takes over 11 seconds to execute when configured loop interval is 5 seconds, causing delayed detection and marking of late flow runs. Execution time exceeds interval by over 2x.

Recommended action:

Review flow_run table query performance and indexes on timestamp columns. Check for growth in active flow runs via prefect.flow_run.count. Monitor prefect.flow_run.late metric for detection delays. Verify indexes on expected_start_time and start_time columns. Consider optimizing late run detection query or increasing loop interval.