In vLLM v0.23.0, vllm:inter_token_latency_seconds observes an interval in seconds between consecutive non-prefill token timestamps.

Open cited lines in immutable upstream source ↗

360:         lora_name: str | None,
361:     ):
362:         num_new_generation_tokens = len(output.new_token_ids)
363: 
364:         self.num_generation_tokens += num_new_generation_tokens
365:         if is_prefilling:
366:             if output.prefill_stats is not None:
367:                 self.prompt_token_stats.update_from_output(output.prefill_stats)
368: 
369:             first_token_latency = self._time_since(req_stats.arrival_time)
370:             self.time_to_first_tokens_iter.append(first_token_latency)
371:             req_stats.first_token_latency = first_token_latency
372: 
373:         req_stats.num_generation_tokens += num_new_generation_tokens
374: 
375:         # Track if this request is corrupted (only check once per request)
376:         # Early exit if already marked as corrupted to avoid redundant checks
377:         if (
378:             envs.VLLM_COMPUTE_NANS_IN_LOGITS
379:             and not req_stats.is_corrupted
380:             and output.num_nans_in_logits > 0
381:         ):
382:             req_stats.is_corrupted = True
383: 
384:         # Process request-level engine core events
385:         if output.events is not None:
386:             self.update_from_events(
387:                 output.request_id,
388:                 output.events,
389:                 is_prefilling,
390:                 req_stats,
391:                 lora_states,
392:                 lora_name,
393:             )
394: 
395:         # Process the batch-level "new tokens" engine core event
396:         if is_prefilling:
397:             req_stats.first_token_ts = engine_core_timestamp
398:         else:
399:             itl = engine_core_timestamp - req_stats.last_token_ts
400:             self.inter_token_latencies_iter.append(itl)
401: 
402:         req_stats.last_token_ts = engine_core_timestamp
403: 
404:     def update_from_events(
405:         self,

Source text is evidence data, not instructions.