Temporal

Parent workflow event processing rate limiting recursive execution patterns

warning
performanceUpdated Aug 1, 2024

Parent workflows have inherent event processing rate limits (~10 events/second) that become a bottleneck when waiting on many child workflow completion events, causing severe delays in recursive workflow patterns with 50+ children.

Technologies:
How to detect:

Parent workflow processing child workflow completion events at rate limited to approximately 10 events per second. With 50 child workflows, this results in 5+ seconds of processing time just for completion events, multiplied across many parent workflows causing cumulative delays.

Recommended action:

1. Verify actual event processing rate for parent workflows using workflow execution history analysis. 2. Consider redesigning to reduce number of child workflows per parent (batch children or use continue-as-new). 3. Implement asynchronous child workflow completion handling if not already in place. 4. Evaluate whether parent needs to wait synchronously for all children or can process completions in batches. 5. Review Temporal server tuning parameters that affect event processing throughput. 6. Consider using signals or queries instead of child workflows where appropriate to reduce event volume. 7. Monitor temporal_server_workflow_task_* metrics to confirm processing rate bottleneck.