Technologies/Langfuse/langfuse.worker.queue.depth
LangfuseLangfuseMetric

langfuse.worker.queue.depth

Background job queue size
Dimensions:None
Available on:PrometheusPrometheus (1)CloudWatchCloudWatch (1)
Interface Metrics (2)
PrometheusPrometheus
Current number of jobs waiting in the background job queue
Dimensions:None
CloudWatchCloudWatch
Number of jobs waiting in the queue
Dimensions:None
Sources
langfuse_background_jobs_queue_sizegithub.com
langfuse.worker.queue.depthlangfuse.com

Technical Annotations (33)

Configuration Parameters (6)
ADMIN_API_KEY
Required for authenticating with BullMQ admin API
LANGFUSE_INGESTION_QUEUE_PROCESSING_CONCURRENCYrecommended: lower than default
Controls parallel job processing; lowering reduces memory spikes at cost of throughput
LANGFUSE_INGESTION_CLICKHOUSE_WRITE_BATCH_SIZErecommended: lower than default
Controls data batched per write; lowering reduces memory consumption
LANGFUSE_TOKEN_COUNT_WORKER_POOL_SIZErecommended: lower than 2 (default)
Tokenizer thread pool; high values cause pending requests to accumulate in memory
NODE_OPTIONSrecommended: --max-old-space-size=6144
Caps Node.js heap to container memory limit, prevents runaway usage
LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MSrecommended: 100
Flushes batches more frequently to prevent large payload buildup
Error Signatures (8)
ERR Error running script (call to f_b588db2e6f108117dcfa5966b343f1203d60c649): @user_script:431: @user_script: 431: -OOM command not allowed when used memory > 'maxmemory'error code
Queue job trace-delete errored: ReplyErrorlog pattern
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memoryexception
Mark-Compactlog pattern
job stalled more than allowable limitexception
Queue job ingestion-job with idlog pattern
504 Gateway Time-outhttp status
HTTPSConnectionPool(host='...'): Read timed out. (read timeout=exception
CLI Commands (7)
curl -X GET "https://your-langfuse-instance.com/api/admin/bullmq" -H "Authorization: Bearer YOUR_ADMIN_API_KEY" -H 'Content-Type: application/json' --data '{"action": "retry", "queueNames": [ "ingestion-queue" ]}'monitoring
curl -X POST "https://your-langfuse-instance.com/api/admin/bullmq" -H "Authorization: Bearer YOUR_ADMIN_API_KEY" -H "Content-Type: application/json" -d '{"action": "retry", "queueNames": ["ingestion-queue", "evaluation-queue"]}'remediation
curl -X POST "https://your-langfuse-instance.com/api/admin/bullmq" -H "Authorization: Bearer YOUR_ADMIN_API_KEY" -H "Content-Type: application/json" -d '{"action": "remove", "queueNames": ["ingestion-queue"], "bullStatus": "failed"}'remediation
ps -eo pid,comm,args | awk "/[n]ode( |$)/ && !/node -e/ {print}"diagnostic
kill -USR1 $PIDdiagnostic
curl -v -X POST https://langfuse.perso.be/api/public/ingestion -H "Content-Type: application/json" -H "Authorization: Basic <base64_creds>" -d '{"batch": [{"type": "trace-create", "id": "test-trace-001", "timestamp": "2025-07-12T12:00:00.000Z", "body": {"id": "test-trace-001", "timestamp": "2025-07-12T12:00:00.000Z", "name": "curl-test"}}]}'diagnostic
curl -v https://langfuse.perso.be/api/public/ingestiondiagnostic
Technical References (12)
BullMQcomponentingestion-queuecomponentevaluation-queuecomponent/api/admin/bullmqcomponentNode.js heapconceptno-eviction cache policyconfigurationobservationsconceptingestion workerscomponentotel directoryfile path/api/public/ingestionendpointnginxcomponentingestion workercomponent
Related Insights (7)
BullMQ queue backlog requires monitoring and managementinfo
▸
Worker container high memory usage with low CPU under ingestion loadcritical
▸
Redis OOM blocks BullMQ job processing and ingestioncritical
▸
High-cardinality traces (10K+ spans) cause Node.js heap exhaustion in ingestion workerscritical
▸
BullMQ ingestion jobs stall when workers are unresponsive under memory pressurecritical
▸
Insufficient capacity amplifies networking issues into cascading failureswarning
▸
504 Gateway Timeout on ingestion endpoint while other API endpoints workcritical
▸