Schema.aivLLM main-2026-07-21 · Version scoped
max_num_batched_tokens
Maximum number of tokens to process in a scheduler iteration.
Settingverified Aug 20
Metric knowledge (2)
| Claim | Semantic layer | Verified |
|---|---|---|
The latest optimization docs snapshot recommends max_num_batched_tokens greater than 8192 for optimal throughput, especially for smaller models on large GPUs. | Interpretation & composition | Aug 20 |
The latest optimization docs snapshot warns that when enable_chunked_prefill=False, max_num_batched_tokens must be greater than max_model_len or vLLM may crash at startup. | Interpretation & composition | Aug 20 |