All versions · Version range: v0.4.2–v0.23.0

max_num_batched_tokens

Maximum number of tokens to process in a scheduler iteration.

SettingvLLM v0.4.2vLLM v0.23.0verified Sep 29

Claim	Semantic layer	Verified
The vLLM v0.4.2 performance docs state that 512 is the default `max_num_batched_tokens` when `enable_chunked_prefill=True`.	Interpretation & composition	Sep 29
In vLLM v0.4.2 `SchedulerConfig`, when `max_num_batched_tokens` is unset and `enable_chunked_prefill=True`, code sets `max_num_batched_tokens` to 512.	Interpretation & composition	Sep 29
In vLLM v0.4.2 `SchedulerConfig`, when `max_num_batched_tokens` is unset and `enable_chunked_prefill=False`, code sets `max_num_batched_tokens` to max(max_model_len, 2048).	Interpretation & composition	Sep 29
In vLLM v0.4.2, `SchedulerConfig` raises an error when `max_num_batched_tokens` is smaller than `max_model_len` and `enable_chunked_prefill=False`.	Interpretation & composition	Sep 29
The latest optimization docs snapshot recommends `max_num_batched_tokens` greater than 8192 for optimal throughput, especially for smaller models on large GPUs.	Interpretation & composition	Aug 20
The latest optimization docs snapshot warns that when `enable_chunked_prefill=False`, `max_num_batched_tokens` must be greater than `max_model_len` or vLLM may crash at startup.	Interpretation & composition	Aug 20

max_num_batched_tokens is exposed as a SchedulerConfig constructor parameter in the v0.4.2 source snapshot.

vllm.config.scheduler_configpartial_promoted_sliceSchedulerConfig constructor surface in vLLM config.py

vLLM v0.4.2 config.py · vllm/config.py:584-608

max_num_batched_tokens is exposed on SchedulerConfig in the v0.23.0 source snapshot.

vllm.config.scheduler_configcomplete_inventorySchedulerConfig dataclass field surface in vllm/config/scheduler.py

vLLM v0.23.0 config scheduler.py · vllm/config/scheduler.py:49-54