Skip to content

Decision record · ADR

ADR-0062: Let the operator choose whether Bench responses stop at end of sequence

Status: accepted | Date: 2026-10-07

References: RFC-0004, ADR-0061

Every serving Bench with a requested output-token limit sends ignore_eos and min_tokens equal to that limit, and RFC-0004:C-INFERENCE-REQUESTS forbids request_body from overriding them. Every response therefore runs to the full limit past its natural end. A downstream SPEED-Bench evaluation of speculative decoding showed the cost: on the qa category, 72% of speculative tokens came after the natural end of sequence, and acceptance length was 3.85 over all tokens against 3.02 up to the first end of sequence. SPEED-Bench’s own methodology, as AIPerf documents it, lets the model stop at end of sequence under an output cap. The operator decided that output length is a measurement choice for downstream users rather than something InferLab forces per benchmark, and that the change must stay compatible with existing definitions, especially the widely used random sources.

Random, random-mixture, and dataset request sources and linear session sources gain one output_stop field. length, the default, keeps today’s behavior: each request’s resolved output-token limit is exact, so InferLab sends min_tokens equal to it and suppresses end of sequence. eos makes the limit a cap: InferLab sends only the maximum and lets the model stop at end of sequence. The field changes only the stop controls a request carries, never the selected samples, prompts, or limits, and the effective value is recorded. It applies uniformly to every source, with no per-benchmark default. Replay sources carry their stop controls in the replayed entries, and the closed AgentX profile keeps the stop controls that make it parameter-identical to InferenceX.

Positive: an operator measuring speculative decoding or natural response lengths can run any request source to its natural end, and nothing in InferLab encodes a benchmark’s methodology. Existing definitions resolve to length and produce byte-identical requests and populations, so their results stay comparable across the upgrade.

Negative: under eos, realized output lengths vary per request and a response may end after one token, so per-token timing covers fewer tokens and throughput comparisons with length runs are not like for like; the recorded output_stop keeps the difference visible. Opting into eos changes a population’s stop controls and therefore its digest. Speculative-decoding acceptance and other output-length-dependent metrics from eos runs are not comparable with length runs, including all existing SPEED-Bench records; compare only runs with the same recorded output_stop.

Neutral: the request-body rule stays in force; min_tokens, min_new_tokens, and ignore_eos remain derived members rather than operator-editable ones.

Add one operator-chosen output_stop to request and session sources, length by default and eos as an opt-in cap (accepted)

Section titled “Add one operator-chosen output_stop to request and session sources, length by default and eos as an opt-in cap (accepted)”
  • Pros: Leaves the measurement choice to the downstream operator for every source, with no benchmark-specific rule, Existing definitions keep their exact requests, populations, and digests
  • Cons: Adds one field to every request and session source, Introduces a second stop-control variant per source, so comparisons must check the recorded output_stop

Make SPEED-Bench always stop at end of sequence, following its methodology (rejected)

Section titled “Make SPEED-Bench always stop at end of sequence, following its methodology (rejected)”
  • Pros: Matches the benchmark’s published methodology without a new field
  • Cons: Hard-codes a per-benchmark rule that goes stale when the benchmark or its tooling changes, Removes a choice operators comparing against exact-length runs need

Allow request_body to set ignore_eos and min_tokens (rejected)

Section titled “Allow request_body to set ignore_eos and min_tokens (rejected)”
  • Pros: No new typed field
  • Cons: Two authorities for one fact: InferLab still derives min_tokens from the output shape, so a body could set ignore_eos=false while min_tokens forces the full length

Keep exact length and record where each response would have ended (rejected)

Section titled “Keep exact length and record where each response would have ended (rejected)”
  • Pros: Keeps one request shape and lets analysis exclude post-EOS tokens
  • Cons: Post-EOS tokens still occupy the batch and KV cache and shape the timing and speculative behavior of every concurrent request, and the server’s acceptance counters aggregate over all generated tokens, so recording the boundary cannot remove post-EOS work from timing or acceptance