Technologies/Prometheus/gunicorn.workers
PrometheusPrometheusMetric

gunicorn.workers

Number of Gunicorn worker processes
Dimensions:None

Technical Annotations (115)

Configuration Parameters (20)
--error-logfilerecommended: /var/log/gunicorn/error.log
enables detailed error logging for startup failures
--preloadrecommended: enabled
loads application code before forking workers to detect import and initialization errors
workersrecommended: 9
For 4-core machine using formula (2 * CPUs) + 1
worker-classrecommended: gevent
Non-blocking worker type for high I/O concurrency in dashboards
threadsrecommended: 1
Threads less critical when using Gevent workers
timeoutrecommended: 60
Worker response timeout in seconds; set higher than longest expected callback
max_requestsrecommended: 1000
restart workers after this many requests to release resources
max_requests_jitterrecommended: 500
randomize worker restarts to avoid thundering herd
worker_classrecommended: sync
sync workers with multiprocessing in app code can cause process proliferation
max-requests
Recycles workers after handling this many requests to prevent memory accumulation
--workersrecommended: 3
number of worker processes for handling concurrent requests
--bindrecommended: unix:/home/ubuntu/my-packing-buddy/myproject.sock
Unix socket path for Gunicorn to bind to
NUM_WORKERSrecommended: tested 1, 3, 9, 17 - no improvement
worker count does not resolve the CPU spike issue
accesslogrecommended: -
Log to stdout for monitoring worker utilization
access_log_formatrecommended: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s" rt=%(L)s busy=%({x-busy}i)s
Include request time (rt=%(L)s) and busy worker count for performance analysis
backlog
controls size of pending connection queue when all workers are busy
preload_apprecommended: True
Required in Gunicorn config when using custom Dockerfile on Cloud Run to eliminate timeout cycle
--timeoutrecommended: 120
Extends worker timeout from default 30s; adjust based on expected request duration
-wrecommended: 4 (or (2 × CPU cores) + 1)
Worker count; insufficient workers cause timeouts under load
--access-logfilerecommended: enabled
Required to expose worker timeout errors in gunicorn logs
Error Signatures (24)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>exception
Worker failed to boot.log pattern
gunicorn.errors.HaltServer: <HaltServer 'worker failed to boot.' 3>exception
Worker with pid 26 was terminated due to signal 9log pattern
Out of memory: Killed process 776660 (gunicorn)log pattern
Worker failed to booterror code
gunicorn.errors.HaltServer: HaltServer 'Worker failed to boot.' 3exception
worker timeoutlog pattern
core dumplog pattern
up == 0error code
RuntimeError: reentrant call inside <_io.BufferedWriter name='<stderr>'>exception
Worker with pid [0-9]+ was terminated due to signal 9log pattern
9error code
Worker with pid %s was terminated due to signal %slog pattern
ImportError: No module namedexception
499http status
WORKER TIMEOUT (pid:log pattern
Worker (pid:*) was sent SIGKILL! Perhaps out of memory?log pattern
upstream prematurely closed connection while reading response header from upstreamlog pattern
502http status
Shutting down: Masterlog pattern
WORKER TIMEOUTlog pattern
[CRITICAL] WORKER TIMEOUT (pid:log pattern
Worker exiting (pid:log pattern
CLI Commands (20)
gunicorn app:app --error-logfile /var/log/gunicorn/error.log --access-logfile /var/log/gunicorn/access.log --preloaddiagnostic
dmesg | grep -i 'killed process'diagnostic
gunicorn app:server --workers $WORKERS --worker-class gevent --bind 0.0.0.0:8000 --timeout 60remediation
pip install gunicorn geventremediation
curl --unix-socket /run/gunicorn.sock http://diagnostic
systemctl status gunicorndiagnostic
systemctl status caddydiagnostic
dmesg -T | egrep -i 'memory'diagnostic
dmesg -T | egrep -i 'oom'diagnostic
gunicorn app:app --workers=4 --max-requests=1000 --max-requests-jitter=100remediation
pkill gunicornremediation
/venv/bin/gunicorn --workers 3 --bind unix:/home/ubuntu/my-packing-buddy/myproject.sock wsgi:appremediation
py-spy --subprocessesdiagnostic
pip install mymoduleremediation
gunicorn --helpdiagnostic
gunicorn --preload mysite.wsgi:applicationdiagnostic
--timeout 120remediation
free -mdiagnostic
--access-logfilediagnostic
-w 4remediation
Technical References (51)
HaltServerexceptionWORKER_BOOT_ERRORerror codeOOM KillercomponentSIGKILLconceptsignal 9conceptWSGI applicationcomponentapp.pyfile pathdotenvcomponentWSGIprotocolGeventcomponentDash callbackscomponentmultiprocessingcomponentOOM killercomponentmemory_profilercomponent/run/gunicorn.sockfile pathsystemd socket activationconceptRDScomponentPrometheus up metricconceptrunbookconceptstderrfile pathlogging deadlockconceptgunicorn.arbitercomponentsync workerscomponentI/O-boundconceptmemcachedcomponentmanage.py runservercomponentconnection backlogconceptupstream_response_timecomponentgunicorn.conf.pyfile pathfaulthandlercomponentworker processcomponentnginxcomponentGoogle App EnginecomponentGoogle Cloud RuncomponentB2 instance classcomponentgthread workercomponentWIFSIGNALEDconceptwaitpidconceptsignal handlercomponentworkercomponentSIGTERMconceptmax_requestsconfigurationmastercomponentpreloadconceptmaster processcomponentload balancercomponenthealth checkconceptsocket accept queueconceptbacklogconceptOS thrashingconceptgunicorn workercomponent
Related Insights (47)
Application code errors prevent Gunicorn worker startupcritical
▸
Insufficient system resources prevent Gunicorn worker initializationcritical
▸
Gunicorn worker terminated by OOM killer due to memory exhaustioncritical
▸
Incorrect WSGI application entry point prevents worker bootcritical
▸
Missing or misconfigured environment variables cause worker startup crashcritical
▸
Missing or incompatible dependencies prevent worker processes from startingcritical
▸
Insufficient server memory prevents worker processes from startingcritical
▸
Circular imports or improper module initialization cause boot failurescritical
▸
Worker pool exhaustion causes exact 60-second request hangscritical
▸
Insufficient worker count for moderate concurrent usage causes request queueingwarning
▸
Gunicorn spawns more processes than configured causing performance degradationwarning
▸
Application memory leak causes OOM after extended uptimecritical
▸
Worker boot timeout triggers core dumps and OOMcritical
▸
Gunicorn unix socket hangs for 2-45 minutes before self-recoverycritical
▸
Multiple Gunicorn servers freeze simultaneously indicating shared resource deadlockcritical
▸
Service down detection requires immediate actioncritical
▸
High worker utilization indicates capacity limit reachedwarning
▸
Low worker count causes request handling degradationwarning
▸
Elevated request duration indicates performance degradationwarning
▸
Reentrant logging deadlock causes worker termination with SIGKILLcritical
▸
Gunicorn workers crash with reentrant logging errors causing intermittent downtimecritical
▸
Insufficient Gunicorn workers cause unpredictable response timeswarning
▸
Old Gunicorn processes block Unix socket preventing restartwarning
▸
Worker timeout prevents hung processes and memory leakswarning
▸
Worker and thread tuning controls memory vs CPU tradeoffinfo
▸
Performance degradation at 400 connections despite low resource utilizationwarning
▸
Excessive worker count degrades performance instead of improving itwarning
▸
Gunicorn CPU spike to 100% causes severe page load delayscritical
▸
Request queueing in connection backlog due to insufficient workerswarning
▸
Requests queue in connection backlog when workers are saturatedwarning
▸
Module import errors prevent Gunicorn worker initializationcritical
▸
Gunicorn configuration errors prevent worker startupcritical
▸
Worker timeout kills process without logging request URIwarning
▸
Worker saturation manifests as nginx 499 status codeswarning
▸
Gunicorn workers enter infinite timeout-SIGKILL cycle on Google App Enginecritical
▸
Worker signal handling failure indicates fatal worker problemcritical
▸
Gunicorn workers OOM killed without logging in Kubernetescritical
▸
Limit worker connections to prevent memory accumulationwarning
▸
Automatic worker restarts cause brief 502 errorswarning
▸
Worker boot failures hidden at default log levelwarning
▸
Worker timeout kills unresponsive workers after 30 secondscritical
▸
Frequent health checks trigger unnecessary worker killswarning
▸
Request duration metric excludes socket accept queue timewarning
▸
Excessive Gunicorn workers cause OS thrashing and performance degradationwarning
▸
Hosting multiple sites on 1GB RAM server causes memory exhaustioncritical
▸
Single Gunicorn worker insufficient for production without benchmarkingwarning
▸
Gunicorn worker timeout causes 20-30 second page delayscritical
▸