Technologies/Celery/celery.worker.up
CeleryCeleryMetric

celery.worker.up

Worker availability status
Dimensions:None
Available on:PrometheusPrometheus (1)
Interface Metrics (1)
PrometheusPrometheus
Worker availability status (1 if up, 0 if down)
Dimensions:None
Sources
celery_worker_upgithub.com

Technical Annotations (48)

Configuration Parameters (11)
spec.template.spec.containers[0].commandrecommended: ["celery"]
Container must run celery command as entrypoint
spec.template.spec.containers[0].argsrecommended: ["-A", "your_app", "worker", "--loglevel=info", "--concurrency=4"]
Arguments to run worker as long-running process
CELERY_BROKER_URLrecommended: amqp://user:pass@rabbitmq:5672/
Required connection string for message broker
CELERY_RESULT_BACKENDrecommended: redis://redis:6379/0
Required connection string for result storage
resources.limits.memoryrecommended: 512Mi
Minimum memory limit for workers
resources.requests.memoryrecommended: 256Mi
Memory request for worker pods
worker_max_memory_per_childrecommended: 200000
Restart worker after consuming 200MB to prevent memory leaks
worker_prefetch_multiplierrecommended: 1
Limit task prefetching to reduce memory pressure
BROKER_URLrecommended: redis://localhost:6379/0
Connection string for message broker - must be accessible from workers
broker_connection_retryrecommended: False
workaround to make worker crash on broker loss, requires restart policy
redisrecommended: 4.5.4 or >=4.6.0
Version 4.5.5 has incompatible Sentinel connection handling
Error Signatures (7)
amqp.exceptions.AccessRefused: ACCESS_REFUSED - Login was refused using authentication mechanism PLAINexception
kombu.connection: Cannot connect to amqp://guest:**@rabbitmq:5672//log pattern
Task requeue attempts exceeded max; marking failedlog pattern
TypeError("SentinelManagedConnection.read_response() got an unexpected keyword argument 'disconnect_on_error'")exception
Unrecoverable error: TypeErrorlog pattern
0exit code
CRITICAL - Unrecoverable errorlog pattern
CLI Commands (14)
kubectl get secret rabbitmq-default-user -n your-namespace -o jsonpath='{.data.username}' | base64 -ddiagnostic
kubectl get secret rabbitmq-default-user -n your-namespace -o jsonpath='{.data.password}' | base64 -ddiagnostic
kubectl get secret celery-broker-secret -n your-namespace -o jsonpath='{.data.BROKER_USERNAME}' | base64 -ddiagnostic
kubectl get secret celery-broker-secret -n your-namespace -o jsonpath='{.data.BROKER_PASSWORD}' | base64 -ddiagnostic
kubectl patch secret celery-broker-secret -n your-namespace --type='json' -p='[{"op": "replace", "path": "/data/BROKER_USERNAME", "value": "'$(kubectl get secret rabbitmq-default-user -n your-namespace -o jsonpath='{.data.username}')'"}]'remediation
kubectl rollout restart deployment/celery-worker -n your-namespaceremediation
kubectl get deployment celery-worker -n your-namespace -o jsonpath='{.spec.template.spec.containers[0].command}'diagnostic
kubectl exec -n your-namespace deployment/celery-worker -- env | grep -iE "celery|redis|rabbit|broker"diagnostic
kubectl top pods -n your-namespace | grep celerymonitoring
kubectl get deployment celery-worker -n your-namespace -o jsonpath='{.spec.template.spec.containers[0].resources}'diagnostic
celery -A proj inspect activemonitoring
celery inspect pingdiagnostic
celery inspect active_queuesdiagnostic
celery -A proj reportdiagnostic
Technical References (16)
FlowercomponentPrometheuscomponentGrafanacomponentSentrycomponentDataDogcomponentCeleryWorkerDownconceptcatatonic stateconceptconsumer registrationconcepttransport levelconceptCeleryExecutorcomponentairflow-providers-celerycomponentrequeue limitconceptSentinelManagedConnectioncomponentSentinelcomponentredis-pycomponentDEBUG level loggingconcept
Related Insights (13)
Authentication failure due to RabbitMQ credential mismatch between secretscritical
▸
Worker pods exit immediately due to incorrect container commandcritical
▸
Worker startup failure due to missing environment variablescritical
▸
Worker OOMKilled due to insufficient memory limitscritical
▸
Inadequate monitoring leads to 65% delayed problem resolutionwarning
▸
Incorrect broker URL configuration prevents task executioncritical
▸
Monitoring tools reduce downtime by 30% through proactive issue detectioninfo
▸
Reduced downtime through continuous monitoring implementationinfo
▸
Celery worker offline for extended period causes task processing haltcritical
▸
Celery worker enters catatonic state after Redis broker restartcritical
▸
Airflow health check fails to detect Celery worker queue consumer losscritical
▸
Celery worker crashes silently with redis-py 4.5.5 when using Sentinelcritical
▸
Silent worker failures masked by exit code 0 require DEBUG loggingwarning
▸