DramatiqRedis

Worker prefetch monopolizes queue in distributed setup

critical
Resource ContentionUpdated Nov 19, 2021(via Exa)
How to detect:

When multiple workers connect to a queue with few messages (e.g., 2-6 messages with 6 workers), the first worker prefetches up to 1000-2000 messages by default, monopolizing all tasks and preventing parallel distribution across workers. Particularly problematic for long-running tasks requiring distribution.

Recommended action:

Set the dramatiq_queue_prefetch environment variable to 1 to limit each worker to prefetching one message at a time per queue. This ensures even task distribution across available workers. Note: default prefetch optimizes for high throughput with fast tasks; reduce only for long-running tasks.