Decision record · ADR
ADR-0028: Keep serving Bench conversations structured through chat completions
Status: superseded | Date: 2026-07-31 Superseded by: ADR-0029
References: RFC-0004, RFC-0005, ADR-0016, ADR-0027, ADR-0025
Context
Section titled “Context”ADR-0016 originally selected chat-completions for serving Bench traffic so the model server applied its chat template. ADR-0025 and ADR-0027 later introduced a workspace template asset and moved every serving Bench to client-rendered completions. The concrete reproduction workflow requires chat-shaped inputs for MTP and structured live history for linear sessions, but it has not demonstrated an operator need to author, compare, or replace arbitrary template assets. The pinned AIPerf chat endpoint already owns message accumulation, live assistant responses, and delayed turns; client rendering would require an additional dynamic renderer or plugin solely to reconstruct behavior the native endpoint already provides. The decision must preserve exact pre-template content-length semantics and request-body controls such as chat_template_kwargs without promoting an unproven template-management surface.
Decision
Section titled “Decision”We will keep every serving Bench request as structured messages and use the pinned AIPerf chat endpoint, leaving the configured model server responsible for applying its model chat template. Bench definitions will not expose a chat-template asset or endpoint switch. This direction wins because it directly matches the independent and linear-session workflows, keeps live session state inside AIPerf, and removes the need for an InferLab-owned renderer or plugin. InferLab will continue to generate and measure message content with the release-pinned tokenizer before server template rendering, while preserving non-structural framework fields such as chat_template_kwargs in request_body. RFC-0004 owns the routing and validation contract, and RFC-0005 owns evidence.
Consequences
Section titled “Consequences”Serving Bench again exercises the same structured chat route used by real conversational traffic, and one-turn populations and dependent sessions share one AIPerf endpoint. Random ISL and dataset admission remain explicit pre-template content measures rather than claims about backend prompt-token counts. InferLab no longer promises byte-identical rendered prompts across different servers or records template bytes; backend qualification and native request artifacts expose the route and messages actually exercised. A server whose model has no usable chat template fails with its own diagnostic, and InferLab does not fall back to completions. Operators cannot use Bench definitions to compare arbitrary template assets; that capability would require a separate concrete workflow and backend transport analysis before it is reconsidered.
Alternatives Considered
Section titled “Alternatives Considered”Keep structured messages on AIPerf chat and use the server model template (accepted)
Section titled “Keep structured messages on AIPerf chat and use the server model template (accepted)”- Pros: Uses AIPerf native live-response and multi-turn behavior, Keeps one request family for independent and dependent conversational Bench traffic
- Cons: Does not promise byte-identical rendered prompts across different server implementations
Resolve and render every chat template in the measurement runtime (rejected)
Section titled “Resolve and render every chat template in the measurement runtime (rejected)”- Pros: Can preserve an exact flat prompt before transport
- Cons: Moves serving Bench away from the native chat endpoint, Requires a dynamic renderer or plugin for live multi-turn history
- Rejected because: No demonstrated operator workflow justifies replacing AIPerf chat with an InferLab-owned renderer.
Expose an optional custom template and derive the route from its presence (rejected)
Section titled “Expose an optional custom template and derive the route from its presence (rejected)”- Pros: Permits explicit template experiments when supported
- Cons: Creates two rendering authorities and two request routes, Introduces backend-specific support before a concrete need exists
- Rejected because: The current reproduction and linear-session workflows need chat formatting, not arbitrary template replacement.