vLLM insights

Open SourceVersions: []57 metrics
BentoML
Prefill stage latency varies wildly with KV-cache layout making baseline modeling noisyinfo

arxiv.org
2mo ago▸
BentoML
torch.compile compilation cache prevents slow cold startswarning

docs.vllm.ai
7d ago▸
BentoML
Auto-tuning disabled by default causes suboptimal kernel performanceinfo

docs.vllm.ai
7d ago▸
BentoML
Dynamic shapes configuration affects guard behavior and performanceinfo

docs.vllm.ai
7d ago▸
BentoML
Cudagraph capture size configuration affects memory and performanceinfo

docs.vllm.ai
7d ago▸