Technologies/MinIO/weaviate_backup_store_to_backend_ms
MinIOMinIOMetric

weaviate_backup_store_to_backend_ms

The number of samples used to calculate the duration of the file transfer stage of a backup store
Dimensions:None

Technical Annotations (49)

Configuration Parameters (15)
GOMEMLIMITrecommended: 10-20% of total memory (e.g., 10GiB for 100GiB nodes)
Controls Go garbage collector soft memory limit; 50GiB on 100GiB nodes (50%) is too high
vectorCacheMaxObjectsrecommended: 500000000000
Half the default value to reduce memory pressure
cpuPercentagerecommended: not specified
Backup parameter to limit CPU usage during backup operations
wait_for_completionrecommended: True
Block until backup/restore completes instead of async operation
BACKUP_S3_BUCKET
S3 bucket name for backup storage; verify bucket exists and is accessible
AWS_REGION
AWS region for S3 bucket; must match bucket region to avoid connectivity issues
chunk_sizerecommended: 256
BackupConfigCreate parameter to control block size, though increasing alone doesn't fix the issue
backups.s3.envconfig.BACKUP_S3_ENDPOINTrecommended: http://minio.default.svc.cluster.local:9000
MinIO service endpoint instead of default AWS S3
backups.s3.envconfig.BACKUP_S3_USE_SSLrecommended: false or true
must match MinIO SSL configuration to prevent connection failures
backups.s3.credentials.AWS_ACCESS_KEY_ID
AWS access key for S3 authentication (method 1)
backups.s3.credentials.AWS_SECRET_ACCESS_KEY
AWS secret key for S3 authentication (method 1)
backups.s3.existingSecret.name
Kubernetes secret name containing AWS credentials (method 2)
backups.gcs.credentials.GOOGLE_APPLICATION_CREDENTIALS
GCP service account JSON for GCS authentication
backups.azure.credentials.AZURE_STORAGE_ACCOUNT
Azure storage account name for authentication
backups.azure.credentials.AZURE_STORAGE_KEY
Azure storage account key for authentication
Error Signatures (3)
upload stat: stat /var/lib/weaviate/document/*/lsm/property__id/primary: no such file or directorylog pattern
BlockListTooLongerror code
The block list may not contain more than 50,000 blockslog pattern
CLI Commands (5)
curl http://localhost:8090/v1/backups/filesystem/my-very-first-backup/restoremonitoring
curl --silent --fail --show-error -H 'Content-Type: application/json' -H "Authorization: Bearer $API_KEY" "http://weaviate.weaviate.svc.cluster.local/v1/backups/backup-s3/$backup_id"diagnostic
POST /v1/backups/s3/{backupId}diagnostic
GET /v1/backups/{provider}monitoring
GET /v1/backups/{provider}/{backupId}monitoring
Technical References (26)
S3 backupcomponentBACKUP_S3_BUCKETconfigurationstatus:STARTEDconceptLSMcomponent/var/lib/weaviatefile pathAzure storage backup modulecomponentbackup-s3component/v1/backups/backup-s3file pathbackup-azurecomponentBackupConfigCreatecomponentshard replica nodescomponentreplicated classescomponentminio-gocomponentmulti-part uploadprotocolMinIOcomponents3.amazonaws.comcomponentKubernetes secretscomponentservice accountconceptRaftprotocolgossip protocolprotocolembedded DBcomponentdata_bind_portcomponentintra-cluster servercomponentasync replicationconceptbackupcomponentv1.26component
Related Insights (13)
Memory increases during S3 backup and never returns to baselinecritical
▸
Asynchronous backup operations require status pollinginfo
▸
Backup fails when LSM property file is missing from filesystemcritical
▸
Long-running backup operations require status monitoringinfo
▸
S3 backup gets stuck in STARTED state indefinitelycritical
▸
Azure backup fails with 50,000 block limit exceededcritical
▸
Backup procedure failed to create shard backups across all replicascritical
▸
Unconfigured MinIO part size causes memory bloat during backupswarning
▸
S3 backup to MinIO fails with default endpoint configurationcritical
▸
Missing cloud provider credentials prevent backup operationscritical
▸
Unmonitored backup operations lead to silent failureswarning
▸
Embedded server backup failure due to Raft port conflictcritical
▸
Async replication pauses during backup operationswarning
▸