Technologies/Weaviate/weaviate_backup_restore_ms
WeaviateWeaviateMetric

weaviate_backup_restore_ms

The number of samples used to calculate the duration of backup restores
Dimensions:None
Available on:DatadogDatadog (1)
Interface Metrics (1)
DatadogDatadog
The number of samples used to calculate the duration of backup restores
Dimensions:None
Sources
weaviate.backup.restore.ms.countgithub.com

Technical Annotations (59)

Configuration Parameters (3)
imagerecommended: semitechnologies/weaviate:<specific-version>
Docker image tag should be pinned version instead of 'latest' to ensure restore compatibility
wait_for_completionrecommended: True
Block until backup/restore completes instead of async operation
CLUSTER_HOSTNAMErecommended: original node hostname from backup source
Required to restore backups on nodes with different hostnames
Error Signatures (16)
entrypoint was deleted in the object store, it has been flagged for cleanup and should be fixed in the next cleanup cyclelog pattern
explorer: get class: vector search: object vector search at indexlog pattern
weaviate.exceptions.UnexpectedStatusCodeExceptionexception
422http status
unable to restore backup as it was produced by a higher versionlog pattern
mmap file: invalid argumenterror code
failed to load shard: Unable to load shardlog pattern
init disk segments: init segmentlog pattern
Error: Restore process failed due to missing files.log pattern
Error: Incomplete restore operation detected.log pattern
resolve node name "0d83d09dea6c" to host'log pattern
coordinator: stat: stat /var/lib/weaviate/.../lsm/property_.../secondary: no such file or directorylog pattern
stat: stat /var/lib/weaviate/.../lsm/property_.../secondary: no such file or directorylog pattern
flush: unlinkat /var/lib/weaviate/test/{index_name}/w8G6ci6OWwEL/lsm/property__id/segment-*.scratch.d: directory not emptylog pattern
write index: unlinkat /var/lib/weaviate/test/{index_name}/w8G6ci6OWwEL/lsm/property_*/segment-*.dbcompaction.scratch.d: directory not emptylog pattern
device or resource busylog pattern
CLI Commands (8)
curl http://localhost:8090/v1/backups/filesystem/my-very-first-backup/restoremonitoring
md5sum /path/to/backup/filediagnostic
weaviate-cli restore --backup /path/to/backup --target /path/to/targetremediation
curl -X POST -H "Content-Type:application/json" <WEAVIATE_URL>/v1/backups/filesystem -d "{\"id\": \"20240712-page_v3\", \"include\": [\"Page_v3\"]}"diagnostic
curl <WEAVIATE_URL>/v1/backups/filesystem/20240712-page_v3diagnostic
find . -name 'secondary'diagnostic
POST /v1/backups/{provider}/{backupId}/restorediagnostic
GET /v1/backups/{provider}/{backupId}monitoring
Technical References (32)
entrypointcomponentshardcomponentobject storecomponentstatus:STARTEDconceptbackup format versionconcepts3componentsemitechnologies/weaviatecomponentclassconceptLSMconceptNFSprotocolAzure file sharecomponent/var/lib/weaviatefile path/var/log/weaviate/file pathCLUSTER_HOSTNAMEcomponentsecondarycomponent.dbcompaction.scratch.dcomponentshard replica nodescomponentreplicated classescomponentclass schemaconceptmulti-tenancyconceptdisaster recoveryconceptRaftprotocolgossip protocolprotocolembedded DBcomponentdata_bind_portcomponentintra-cluster servercomponentLSM (Log-Structured Merge-tree)componenttombstonesconceptmemtable flushcomponentcompactioncomponentscratch spacecomponentfilesystemcomponent
Related Insights (17)
Vector search entrypoint deleted after backup restore cyclecritical
▸
Backup restore processes fail when actually needed in productioncritical
▸
Asynchronous backup operations require status pollinginfo
▸
Weaviate remains operational during backup and restore operationsinfo
▸
Backup version mismatch prevents restorationcritical
▸
Backup restore completes with success status but transfers only schema when versions mismatchcritical
▸
Backup restoration fails with mmap error due to LSM segment corruption on NFS storagecritical
▸
Weaviate restore process fails to completecritical
▸
Long-running backup operations require status monitoringinfo
▸
Backup restore fails with hostname resolution error on different nodecritical
▸
Backup fails when LSM compaction scratch directory exists with missing secondary filecritical
▸
Backup procedure failed to create shard backups across all replicascritical
▸
Incomplete backup of class schemas on nodes without local datacritical
▸
Untested backup restore fails during disaster recoverycritical
▸
Embedded server backup failure due to Raft port conflictcritical
▸
Backup failure after bulk object deletion due to LSM compaction conflictscritical
▸
Cross-version backup restore causes tenant data inaccessibilitycritical
▸