Technologies/Istio/envoy.server.memory.heap
IstioIstioMetric

envoy.server.memory.heap

Heap memory size in bytes
Dimensions:None

Technical Annotations (61)

Configuration Parameters (12)
dynamic_resources.cds_config.initial_fetch_timeoutrecommended: 0s
Zero timeout used in reproduction config; non-zero may change behavior
dynamic_resources.lds_config.initial_fetch_timeoutrecommended: 0s
Zero timeout used in reproduction config; non-zero may change behavior
lds_config.api_config_source.api_typerecommended: DELTA_GRPC
Enables incremental listener discovery to reduce config update overhead
cds_config.api_config_source.api_typerecommended: DELTA_GRPC
Enables incremental cluster discovery to reduce config update overhead
dynamic_resources.lds_config.api_config_source.api_typerecommended: DELTA_GRPC
Enables delta updates for listener discovery instead of full config updates
dynamic_resources.cds_config.api_config_source.api_typerecommended: DELTA_GRPC
Enables delta updates for cluster discovery instead of full config updates
dynamic_resources.lds_config.api_config_source.transport_api_versionrecommended: V3
Specifies xDS transport API version for listener config
dynamic_resources.cds_config.api_config_source.transport_api_versionrecommended: V3
Specifies xDS transport API version for cluster config
spec.featureFlags.enableWASMStatsrecommended: true
Enables metrics collection but contributes to memory issue when combined with permissive mode
spec.traffic.enablePermissiveTrafficPolicyModerecommended: false
Setting to false prevents polynomial route growth and controls memory consumption
initial_stream_window_size
HTTP/2 initial stream window size; reduce from large defaults
initial_connection_window_size
HTTP/2 initial connection window size; reduce from large defaults
Error Signatures (6)
grpc-status: 14error code
no healthy upstreamlog pattern
gRPC config stream to cluster1 closed sincelog pattern
gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failedlog pattern
Unable to establish new stream to configuration serverlog pattern
Envoy is not fully initialized, skipping histogram merge and flushing statslog pattern
CLI Commands (13)
kubectl top pod my-service-pod -n my-namespace --containersdiagnostic
kubectl exec deploy/my-service -c istio-proxy -- pilot-agent request GET stats | grep "server.memory"diagnostic
kubectl get pod my-service-pod -n my-namespace -o jsonpath='{.spec.containers[?(@.name=="istio-proxy")].resources.limits.memory}'diagnostic
kubectl exec deploy/my-service -c istio-proxy -- pilot-agent request GET stats | grep "server.memory_allocated"diagnostic
kubectl exec deploy/my-service -c istio-proxy -- pilot-agent request GET stats | grep "server.memory_heap_size"diagnostic
ps_memdiagnostic
topdiagnostic
smemdiagnostic
curl 127.0.0.1:19000/memorymonitoring
curl 127.0.0.1:19000/heap_dump -o memory-bug.heapdiagnostic
pprof -http localhost:9999 memory-bug.heapdiagnostic
diff <(jq --sort-keys . envoy-before.json) <(jq --sort-keys . envoy-after.json)diagnostic
curl -s localhost:9901/server_infodiagnostic
Technical References (30)
istio-proxycomponentsidecarcomponentRSS (resident set size)conceptPSS (Proportional Set Size)conceptUSS (Unique Set Size)conceptserver.memory_allocatedcomponentserver.memory_heap_sizecomponentTlsCachingDateProviderImplcomponentTimerImplcomponentxDS protocolprotocolADS (Aggregated Discovery Service)componentthread local storageconceptxDSprotocolLDScomponentCDScomponentproto_descriptor_bincomponentgRPC JSON transcoder filtercomponentfilter chainscomponenthttp 1.1 parsercomponentbuffering filtercomponentDELTA_GRPCprotocollds_configcomponentcds_configcomponentdynamic_resourcescomponentHTTPRouteGroupcomponentTrafficTargetcomponentOpen Service Meshcomponentenvoy sidecarcomponenthttp2_settingscomponent/statscomponent
Related Insights (11)
Envoy sidecar OOM kill causes connection drops and service outagescritical
▸
Memory fragmentation indicated by heap size vs allocated gapwarning
▸
Memory exhaustion causes OOMKills and proxy restartscritical
▸
Envoy internal memory metrics underreport actual process memory usagewarning
▸
Memory leak when xDS management server unavailable at startup with static resourcescritical
▸
Memory spikes during pod scaling due to full Envoy config updateswarning
▸
gRPC JSON transcoder proto descriptors cause 2x memory during listener updateswarning
▸
HTTP/1.1 parser packet data accumulation causes OOMcritical
▸
Full configuration updates cause memory spikes during pod scalingwarning
▸
Polynomial memory consumption with WASM stats and permissive traffic policy enabledcritical
▸
Default HTTP/2 window sizes cause excessive memory usagewarning
▸