Technologies/Weaviate/weaviate_async_operation_running
WeaviateWeaviateMetric

weaviate_async_operation_running

The number of currently running async operations. The operation itself is defined through the operation label
Dimensions:None
Available on:DatadogDatadog (1)
Interface Metrics (1)
DatadogDatadog
The number of currently running async operations. The operation itself is defined through the operation label
Dimensions:None
Sources
weaviate.async.operations.runninggithub.com

Technical Annotations (59)

Configuration Parameters (8)
ASYNC_REPLICATION_PROPAGATION_CONCURRENCYrecommended: 1
reduces async replication load to allow RAFT participation during bootstrap
asyncEnabledrecommended: true
Enables async replication but gets reset to false on restart for migrated classes
imagerecommended: semitechnologies/weaviate:<specific-version>
Docker image tag should be pinned version instead of 'latest' to ensure restore compatibility
wait_for_completionrecommended: True
Block until backup/restore completes instead of async operation
BACKUP_S3_BUCKET
S3 bucket name for backup storage; verify bucket exists and is accessible
AWS_REGION
AWS region for S3 bucket; must match bucket region to avoid connectivity issues
ASYNC_REPLICATION_PROPAGATION_TIMEOUTrecommended: 1m (default), increase for large objects/high latency
Maximum time for propagation request to remote node
ASYNC_REPLICATION_PROPAGATION_BATCH_SIZErecommended: reduce for large objects
Number of objects per batch affects propagation time
Error Signatures (15)
'coroutine' object has no attribute 'multi_tenancy_config'exception
coroutine '_ConfigCollectionAsync.get' was never awaitedlog pattern
file existserror code
restore class ExampleClass: write files: move files to destinationlog pattern
rename /var/lib/weaviate/.backup.tmp/log pattern
weaviate.exceptions.UnexpectedStatusCodeExceptionexception
422http status
unable to restore backup as it was produced by a higher versionlog pattern
Error: Restore process failed due to missing files.log pattern
Error: Incomplete restore operation detected.log pattern
backup .* already in progresslog pattern
cannot create new backup, backup 'weaviate-backup_20240902600000' is not yet releasedlog pattern
this means its contents have not yet been fully copied to its destination, try again laterlog pattern
hashbeat iteration failed: collecting differenceslog pattern
context deadline exceededexception
CLI Commands (6)
curl http://localhost:8090/v1/backups/filesystem/my-very-first-backup/restoremonitoring
md5sum /path/to/backup/filediagnostic
weaviate-cli restore --backup /path/to/backup --target /path/to/targetremediation
curl --silent --fail --show-error -H 'Content-Type: application/json' -H "Authorization: Bearer $API_KEY" "http://weaviate.weaviate.svc.cluster.local/v1/backups/backup-s3/$backup_id"diagnostic
curl --silent --fail --show-error -X POST -H 'Content-Type: application/json' -H "Authorization: Bearer $API_KEY" "http://weaviate.weaviate.svc.cluster.local/v1/backups/s3" -d "{\"id\": \"$backup_id\"}"remediation
DELETE /backups/{backend}/{id}remediation
Technical References (30)
RAFTprotocolhashbeatcomponentWeaviateAsyncClientcomponentWeaviateVectorStorecomponentinsert_manycomponent_ConfigCollectionAsync.getcomponentstatus:STARTEDconcept.hnsw.commitlog.dfile path/var/lib/weaviate/.backup.tmpfile path/var/lib/weaviatefile pathGCS backendcomponentbackup format versionconcepts3componentsemitechnologies/weaviatecomponent/var/log/weaviate/file pathbackup-s3component/v1/backups/backup-s3file pathtry_backupcomponent/backups/{backend}/{id}componenthashtreecomponentasync_replicationcomponentpropagation phaseconceptRaftprotocolgossip protocolprotocolembedded DBcomponentdata_bind_portcomponentintra-cluster servercomponentasync replicationconceptbackupcomponentv1.26component
Related Insights (17)
Async replication overload during bootstrap blocks RAFT participationwarning
▸
Async indexing experimental status may cause stability issueswarning
▸
LangChain WeaviateVectorStore fails with async client due to unawaited coroutinewarning
▸
Async replication fails for classes migrated from 1.24.x to 1.26.xcritical
▸
Asynchronous backup operations require status pollinginfo
▸
Weaviate remains operational during backup and restore operationsinfo
▸
Multiple restores from same backup ID fail with file exists errorwarning
▸
Backup version mismatch prevents restorationcritical
▸
Weaviate restore process fails to completecritical
▸
Long-running backup operations require status monitoringinfo
▸
S3 backup gets stuck in STARTED state indefinitelycritical
▸
Concurrent backup attempts fail with 422 error when backup in progresswarning
▸
Failed backup prevents all subsequent backups from startingcritical
▸
Async replication hashbeat fails with context deadline exceeded in geo-distributed clusterscritical
▸
Propagation timeouts with large objects or high network latencywarning
▸
Embedded server backup failure due to Raft port conflictcritical
▸
Async replication pauses during backup operationswarning
▸