Task run count API endpoint blocks UI with millions of task runs
criticalperformanceUpdated Apr 24, 2025(via Exa)
How to detect:
The /runs UI page becomes unusable when api/task_runs/count query takes excessive time to return counts in the millions (e.g., 6.3M task runs). The entire UI hangs waiting for the count, even when the database CPU is at 20-40% utilization.
Recommended action:
Enable the database vacuum background service (off by default) to prevent data accumulation. Implement aggressive data retention policies using client.delete_flow_run() or direct table cleanup for task_run, task_run_state tables followed by VACUUM ANALYZE. Consider whether real-time counts are necessary or if approximate/cached counts would suffice. Upgrade to versions with improved task_run indexing (e.g., PR #20940).