MarkLateRuns service execution exceeds loop interval
warningperformanceUpdated Apr 8, 2025(via Exa)
Technologies:
How to detect:
MarkLateRuns service takes over 11 seconds to execute when configured loop interval is 5 seconds, causing delayed detection and marking of late flow runs. Execution time exceeds interval by over 2x.
Recommended action:
Review flow_run table query performance and indexes on timestamp columns. Check for growth in active flow runs via prefect.flow_run.count. Monitor prefect.flow_run.late metric for detection delays. Verify indexes on expected_start_time and start_time columns. Consider optimizing late run detection query or increasing loop interval.