Message duplication from RabbitMQ connection closure under CPU-intensive load
criticalWhen running CPU-intensive tasks with high thread count (e.g., 40 threads), consumer threads become starved and cannot communicate with RabbitMQ. This prevents heartbeat exchanges, causing RabbitMQ to close connections and automatically unack messages in flight. Upon reconnection, duplicate messages appear across queues, potentially more than doubling the original message count.
1) Reduce worker thread count to prevent consumer thread starvation. 2) Increase RabbitMQ heartbeat timeout to allow longer intervals between communication. 3) Refactor CPU-intensive tasks to introduce explicit pauses or yield points. 4) Lower Python's sys.setswitchinterval to increase thread switching frequency. 5) Ensure all tasks are idempotent since Dramatiq provides at-least-once delivery semantics.