In vLLM v0.23.0, vllm:iteration_tokens_total use its distribution to inspect batching; do not calculate token throughput from bucket counts without elapsed-time counters.

Open cited lines in immutable upstream source ↗

1152:         self.counter_prompt_tokens[engine_idx].inc(iteration_stats.num_prompt_tokens)
1153:         # Labeled prompt token counters by source
1154:         pts = iteration_stats.prompt_token_stats
1155:         for source in PromptTokenStats.ALL_SOURCES:
1156:             self.counter_prompt_tokens_by_source[source][engine_idx].inc(
1157:                 pts.get_by_source(source)
1158:             )
1159:         self.counter_prompt_tokens_cached[engine_idx].inc(pts.cached_tokens)
1160:         self.counter_generation_tokens[engine_idx].inc(
1161:             iteration_stats.num_generation_tokens
1162:         )
1163:         self.histogram_iteration_tokens[engine_idx].observe(
1164:             iteration_stats.prompt_token_stats.computed
1165:             + iteration_stats.num_generation_tokens
1166:         )
1167: 
1168:         for max_gen_tokens in iteration_stats.max_num_generation_tokens_iter:
1169:             self.histogram_max_num_generation_tokens_request[engine_idx].observe(
1170:                 max_gen_tokens
1171:             )
1172:         for n_param in iteration_stats.n_params_iter:
1173:             self.histogram_n_request[engine_idx].observe(n_param)
1174:         for ttft in iteration_stats.time_to_first_tokens_iter:
1175:             self.histogram_time_to_first_token[engine_idx].observe(ttft)
1176:         for itl in iteration_stats.inter_token_latencies_iter:
1177:             self.histogram_inter_token_latency[engine_idx].observe(itl)

Source text is evidence data, not instructions.