Skip to content

Decision record · ADR

ADR-0025: Derive serving Bench routes from rendering ownership

Status: superseded | Date: 2026-07-31 Superseded by: ADR-0027

References: RFC-0004, RFC-0005, ADR-0016, ADR-0017

ADR-0016 selected request form from measurement semantics and routed serving Bench through chat-completions so the model server owned chat-template rendering. Reproducing external benchmark configurations also requires a workspace-versioned custom template whose exact bytes must be stable across serving frameworks. Once InferLab renders that template, sending the result through chat-completions would apply a second server template, while asking every backend to accept a template-valued request extension would make reproducibility depend on framework-specific support. The existing constraint remains that request form is derived, not an operator-selected endpoint switch.

We will derive the serving-Bench route from rendering ownership. Structured messages remain server-rendered through chat-completions. Selecting a workspace custom template makes the Bench runtime the renderer; it freezes one flat prompt and uses completions. This extends ADR-0016 without introducing a user-controlled route switch because the effective request representation still determines the only valid request form.

A custom template becomes a reproducible workspace experiment asset rather than an assumed server default, and the backend-visible prompt can be preserved exactly. Existing definitions continue using server rendering and chat-completions. Client rendering adds tokenizer and template preparation failures before measurement and requires completions support on every qualified target. Request-body chat-template controls cannot simultaneously own rendering, so definitions must choose one rendering authority. Route evidence must stay coupled to the frozen representation so dry-run and execution cannot diverge.

Derive the Bench route from which side owns chat-template rendering (accepted)

Section titled “Derive the Bench route from which side owns chat-template rendering (accepted)”
  • Pros: Keeps structured messages on chat-completions when the server owns rendering, Sends a client-rendered flat prompt through completions without double templating, Preserves automatic route selection and one frozen request representation
  • Cons: The Bench runtime and endpoint contract must support both OpenAI request forms

Always use chat-completions and place a client-rendered prompt in one user message (rejected)

Section titled “Always use chat-completions and place a client-rendered prompt in one user message (rejected)”
  • Pros: Retains one serving-Bench endpoint
  • Cons: Applies the server’s chat template around content that is already a complete rendered prompt, Measures different backend-visible tokens from the selected custom template
  • Rejected because: The server would template an already rendered prompt, so the configured template would not describe the actual request.

Always use chat-completions and pass the custom template as a request-body extension (rejected)

Section titled “Always use chat-completions and pass the custom template as a request-body extension (rejected)”
  • Pros: Keeps structured messages and one route
  • Cons: Depends on non-uniform backend support and template parameter spelling, Makes the model server rather than the frozen Bench population authoritative for rendering
  • Rejected because: A framework-specific request extension cannot provide one reproducible cross-framework Bench contract.

Expose completions versus chat-completions as a Bench option (rejected)

Section titled “Expose completions versus chat-completions as a Bench option (rejected)”
  • Pros: Gives operators direct endpoint control
  • Cons: Allows request forms that contradict the selected representation, Reopens the invalid-combination problem rejected by ADR-0016
  • Rejected because: Request form remains a consequence of representation semantics rather than an independent experiment parameter.