Technologies/Redis/spinnaker.clouddriver.cache.refresh.duration

RedisMetric

spinnaker.clouddriver.cache.refresh.duration

Cache refresh duration

Dimensions:None

Technical Annotations (47)

Configuration Parameters (8)

redis.poll.timeoutSecondsrecommended: increase from 300

Timeout before rescheduling agents that haven't completed (default 5 minutes)

redis.poll.intervalSecondsrecommended: 30

Default 30-second interval between caching agent runs; tune based on freshness vs quota needs

cacheIntervalSecondsrecommended: 300

controls frequency of kubernetes resource caching, default 30s is too aggressive for large clusters

clouddriver.cache.timeoutrecommended: explicit value needed

Not set by default; explicit configuration prevents cache expiration causing visibility loss

clouddriver.resources.limits.cpurecommended: 4000m-8000m for caching component

For deployments managing 9+ K8s clusters plus EC2 regions

clouddriver.resources.limits.memoryrecommended: 6Gi-12Gi for caching component

For deployments managing 9+ K8s clusters plus EC2 regions

cacheThreadsrecommended: 10-20

Per-account setting in clouddriver-local.yml to parallelize cache operations for large clusters

redis.poll.errorIntervalSecondsrecommended: 60

Error polling interval for Redis caching agent

Error Signatures (7)

Force cache refresh response narrowed to ... was null, and this group hasn't been processed yet!log pattern

No recent pending force cache refresh updates found, retrying in 10slog pattern

java.lang.NullPointerException: Cannot invoke method countOnDemand() on null objectexception

failed to handle on demand update for ServerGrouplog pattern

Pending refresh of ManifestForceCacheRefreshTasklog pattern

was cached as a part of this request, but not processedlog pattern

knownInstanceIds: [null]log pattern

CLI Commands (2)

curl -H 'content-type:application/json' 'http://<clouddriver endpoint>/cache/aws/ServerGroup' -d '{"asgName":<asg name>,"serverGroupName":<server group name>,"region":<region>,"account":<account>}'

diagnostic

while true; do curl -H 'X-SPINNAKER-USER: <username>' 'http://<clouddriver endpoint>/applications/<application>/clusters/<account>/<cluster>/aws/serverGroups/<server group>?region=<region>' | jq '[.instances[].id]'; sleep 1; done

diagnostic

Technical References (30)

clouddrivercomponentelasticachecomponenthalyardcomponentClouddrivercomponentcaching agentsconcept~/.hal/$DEPLOYMENT/profiles/clouddriver-local.ymlfile pathKubernetesCachingAgentcomponentKubernetesCoreCachingAgentcomponentKubernetesCacheDataConvertercomponentServerGroupCacheForceRefreshTaskcomponentOrcacomponent/cache/kubernetes/ServerGroupfile pathForce Cache Refreshconcepton-demand cache refreshconceptclouddriver-cachingcomponentclouddriver-rwcomponentclouddriver-rocomponentinfrastructure tabcomponentCatsOnDemandCacheUpdatercomponentOnDemandMetricsSupportcomponentKubernetesV1Providercomponentclouddriver-local.ymlfile pathred/black deployment strategyconceptorcacomponentManifestForceCacheRefreshTaskcomponentClusterCachingAgentcomponentauthoritativeTypesconceptputCacheResultcomponent/cache/aws/ServerGroupfile pathImageCachingAgentcomponent

Related Insights (13)

ElastiCache Redis causes stale clouddriver account configuration and pipeline failurescritical

▸

Premature caching agent rescheduling due to timeoutwarning

▸

Caching interval tradeoff between UI freshness and API quotainfo

▸

KubernetesCachingAgent processes excessive manifest relationships in tight loopswarning

▸

Clouddriver prematurely evicts on-demand cache requests causing extended Force Cache Refresh timeswarning

▸

Single Clouddriver pod replica creates single point of failurewarning

▸

Clouddriver cache timeout causes clusters to disappear from infrastructure tabcritical

▸

Namespaces disappear from UI requiring clouddriver-caching restartwarning

▸

NullPointerException in cache metrics prevents on-demand cache updatescritical

▸

Cache refresh performance degradation with large Kubernetes clusterswarning

▸

Force Cache Refresh stage pending delays Kubernetes deploymentswarning

▸

On-demand ServerGroup cache refresh causes temporary instance data losscritical

▸

Clouddriver caches large resource sets in memory despite external cache backendwarning

▸