Technologies/Redis/dramatiq.messages.retried

RedisMetric

dramatiq.messages.retried

Total messages retried

Dimensions:None

Technical Annotations (51)

Configuration Parameters (16)

fail_fastrecommended: True

StubBroker.join parameter to reraise exceptions in main thread (default since v2.0.0)

middlewarerecommended: []

Pass to broker constructor to prevent default middlewares from loading

max_retriesrecommended: 3

Set on Retries middleware when explicitly added; value depends on use case

on_failurerecommended: on_task_failure callback

Captures permanently failed tasks for alerting and dead letter queue

min_backoffrecommended: 1000

Initial retry delay in milliseconds (1000 = 1 second)

max_backoffrecommended: 60000 to 300000

Maximum retry delay in milliseconds (60000=1min, 300000=5min)

time_limitrecommended: 24 hours

allows actor to run for extended duration without timeout

heartbeat_timeoutrecommended: 300000

increased timeout in RedisBroker constructor to prevent premature worker death detection

dramatiq_queue_prefetchrecommended: 1

limits to one job per worker/process to prevent multiple duplicate executions

broker.heartbeatrecommended: 30s

prevents worker desync when network latency is high

max_delayrecommended: 3600

cap in seconds to prevent multi-day retry delays

idemp:{msg_id}recommended: 24h TTL

Redis key pattern for message deduplication tracking

prefetch_multiplierrecommended: 1

prevents worker starvation in retry-heavy workloads

consumer-timeoutrecommended: 30 minutes (default)

RabbitMQ timeout for message acknowledgment before connection closure

TimeLimitrecommended: 10 minutes (default, must be < consumer-timeout)

Dramatiq middleware enforcing maximum task execution time

requeuerecommended: True

In basic_nack call to enable message redelivery on actor failures

Error Signatures (4)

response.raise_for_status()exception

azure.core.exceptions.ResourceNotFoundError: The specified message does not existexception

MessageNotFounderror code

Unhandled error during post_process_messagelog pattern

CLI Commands (2)

py.test --log-cli-level=warningdiagnostic

broker.declare_queue('dlq', durable=True)remediation

Technical References (29)

StubBrokercomponentRetries middlewarecomponentwork queueconcepttime.sleepcomponentkafka providercomponentRedisBrokercomponentRetriescomponentdefault_middlewarescomponentidempotentconceptdead_letter_queuecomponentretry_countconcept__heartbeats__component__acks__componentdispatch.luacomponentConcurrentRateLimitercomponentRedisBackendcomponentmessage_idcomponentmsg_idcomponentsupplier_tx_hashcomponentdead letter queuecomponentDLQcomponentTimeLimit middlewarecomponentdramatiq-azurecomponentpost_process_messagecomponent/usr/local/lib/python3.11/dist-packages/dramatiq/worker.pyfile path/usr/local/lib/python3.11/dist-packages/dramatiq_azure/asq.pyfile pathNACKconceptbasic_nackcomponent@dramatiq.actorcomponent

Related Insights (21)

Integration tests hang when actors fail due to retry exhaustion delaywarning

▸

Worker pool exhaustion from blocking retry delays prevents message processingcritical

▸

Millisecond to second conversion bug causes 1000x longer retry delayscritical

▸

Duplicate Retries middleware causes retry count multiplicationwarning

▸

Worker failure causes duplicate message delivery requiring idempotent actorscritical

▸

Tasks fail permanently after exhausting retriescritical

▸

Task fails permanently after retry exhaustioncritical

▸

Exponential backoff delays reach maximum causing extended retry intervalsinfo

▸

External API HTTP errors trigger task retrieswarning

▸

Messages re-queued despite valid heartbeat during idle waitwarning

▸

Concurrent rate limiters trigger retry stormswarning

▸

Network latency above 200ms causes worker desync and phantom retriescritical

▸

Exponential backoff overflow causes multi-day delays at high retry countswarning

▸

Non-idempotent tasks with retries cause duplicate processing and data corruptioncritical

▸

Worker starvation under high retry load with default prefetch settingswarning

▸

High retry rates degrade throughput by 40% across all queue systemswarning

▸

Missing dead letter queue causes infinite retry resource exhaustionwarning

▸

RabbitMQ consumer timeout causes infinite message reprocessingcritical

▸

Azure Queue message deletion failure causes duplicate processingcritical

▸

NACK hardcoded to not requeue sends failed messages to DLQ instead of retryingwarning

▸

Task duration exceeding time_limit triggers unwanted retrieswarning

▸