Missing 'Received message' log indicates worker pool exhaustion
warningavailabilityUpdated Sep 26, 2024(via Exa)
Technologies:
How to detect:
When a message shows 'Pushing message onto work queue' log at worker.py:327 but never shows the corresponding 'Received message' log at worker.py:481, it indicates the worker pool is exhausted and unable to process queued messages.
Recommended action:
Search logs for messages with only 'Pushing message' log at worker.py:327. Check if 'Received message' log at worker.py:481 appears for same message ID. If missing, investigate worker pool capacity using dramatiq.worker.idle and dramatiq.worker.busy metrics. Check for blocking operations or long-running tasks holding workers.