Skip to content

Decision record · ADR

ADR-0027: Freeze serving Bench chat templates in the measurement runtime

Status: superseded | Date: 2026-07-31 Superseded by: ADR-0028

References: RFC-0004, RFC-0005, ADR-0016, ADR-0025

ADR-0016 selected request form from measurement semantics and sent serving Bench traffic through chat-completions, while ADR-0025 later split rendering ownership: omission left the model server in control and a workspace template moved rendering into the Bench runtime. The confirmed operator workflow instead chooses one chat template before a Bench starts, either the model tokenizer default or a supplied workspace asset, and keeps that template fixed for every request. A linear session changes its live message history after each response, but it does not change template identity. Keeping two rendering authorities makes omission alter both prompt ownership and protocol route, and it leaves the actual default prompt dependent on backend deployment state. The decision must preserve task-directed Eval routes while making serving Bench prompts comparable across qualified frameworks without exposing an endpoint switch.

We will resolve every serving Bench chat template in the release-owned measurement runtime before execution. The model tokenizer default and a workspace-supplied asset are two provenance sources for the same frozen template representation. The runtime renders each independent request or current linear-session history with that frozen representation and uses completions for the resulting flat prompt. This direction wins because template selection no longer changes rendering ownership, and the backend-visible prompt is fixed independently of a serving framework’s default chat handling. The task-directed portions of ADR-0016 remain: generative lm-eval tasks use their structured chat path, likelihood tasks use completions, and the built-in smoke uses completions. Request form remains derived from measurement semantics rather than becoming an operator-selected endpoint control. This decision replaces the split rendering model in ADR-0025 and the serving-Bench portion of ADR-0016; it does not choose an AIPerf extension or integration mechanism. RFC-0004 remains authoritative for routing, validation, and compatibility behavior, while RFC-0005 remains authoritative for evidence.

Serving Bench gains one template and prompt authority across default and custom cases, so SGLang, vLLM, and other qualified targets receive the same rendered prompt for the same frozen inputs. Independent requests can freeze their flat prompts during preparation, while linear sessions apply the same frozen template to live history as it develops. Every serving Bench now pays tokenizer and template preparation cost and requires a qualified completions route; a missing usable tokenizer default becomes a preparation failure instead of being deferred to a chat server. An existing definition that omits chat_template keeps the same syntax but changes rendering authority, backend-visible prompt, and route from server-rendered chat-completions to client-rendered completions. Bench no longer exercises a backend’s chat-template implementation, while generative Eval continues to cover the structured chat route. Non-template framework-specific request fields still cross unchanged and can be rejected by a completions endpoint; the owning RFC defines conflicts with server-side template controls. The concrete way the pinned AIPerf runtime receives a dynamic session renderer remains an integration concern that requires separate evidence and must not become a public Bench plugin selector.

Resolve either the tokenizer default or one workspace template before Bench execution and always render in the measurement runtime (accepted)

Section titled “Resolve either the tokenizer default or one workspace template before Bench execution and always render in the measurement runtime (accepted)”
  • Pros: Makes the actual template and backend-visible prompt reproducible across serving frameworks, Keeps default and custom selection as provenance variants of one request representation
  • Cons: Requires every Bench target to support completions and every Bench run to prepare the tokenizer

Keep default templates server-rendered and custom templates client-rendered (rejected)

Section titled “Keep default templates server-rendered and custom templates client-rendered (rejected)”
  • Pros: Uses native chat-completions for definitions without a custom asset
  • Cons: Makes omission change both rendering authority and request family, Leaves the effective default template dependent on backend deployment state
  • Rejected because: One product workflow should not change its rendering authority merely because the operator omitted an override asset.

Always use server rendering and pass custom templates through framework request extensions (rejected)

Section titled “Always use server rendering and pass custom templates through framework request extensions (rejected)”
  • Pros: Retains structured messages and chat-completions for all Bench requests
  • Cons: Depends on non-standard extension names and backend-specific template handling
  • Rejected because: A framework-specific request extension cannot freeze one comparable cross-backend Bench prompt.

Expose completions versus chat-completions as an operator-selected Bench option (rejected)

Section titled “Expose completions versus chat-completions as an operator-selected Bench option (rejected)”
  • Pros: Allows operators to choose either protocol directly
  • Cons: Creates request forms that can contradict the selected rendering authority
  • Rejected because: The request family is a consequence of rendering and task semantics rather than an independent experiment control.