Skip to content

Decision record · ADR

ADR-0016: Select OpenAI request form from measurement semantics

Status: superseded | Date: 2026-07-15 Superseded by: ADR-0027

References: RFC-0003, RFC-0004, RFC-0006

InferLab currently records one public completions path and sends lm-eval and serving-Bench traffic through raw completions. That shape cannot express model chat templates or request-scoped reasoning controls, while replacing every request with chat-completions would remove the prompt-logprob semantics required by likelihood evaluation.

InferLab needs one reproducible definition-layer authority for real inference request parameters while preserving the different protocol requirements of generative tasks, likelihood scoring, and the built-in endpoint smoke. The same decision must remain valid across direct serving, native routers, and InferLab-owned prefill/decode proxies for vLLM, SGLang, TensorRT-LLM, and TokenSpeed.

RFC-0003 keeps public routing and proxy lifecycle in the serving contract. RFC-0004 keeps task, prompt, output type, and scoring semantics in lm-eval while InferLab owns measurement orchestration. RFC-0006 makes integrations authoritative for framework endpoint contracts. Prompt loglikelihood requires the completions request and echoed prompt-token log probabilities. Reasoning controls such as reasoning_effort and chat_template_kwargs are real chat request fields whose supported values remain framework- and model-specific.

The discussion considered selecting request form from task semantics, using chat-completions for every request, exposing request form as a definition switch, and normalizing framework-specific inference controls into an InferLab-owned profile.

We will select the OpenAI request form from resolved measurement semantics and preserve definition-scoped request fields as an exact request-body fragment because:

  1. Protocol fidelity: generate_until and serving performance requests need chat-completions to exercise the model real chat template and reasoning controls, while likelihood scoring and the built-in smoke require completions semantics.
  2. Single authority: One Eval definition selects one individual lm-eval task, which owns prompt or message construction, output bounds, stops, and request form; the measurement definition owns the remaining declared body fragment. Recipes compose multiple Eval definitions when an experiment needs multiple tasks.
  3. Framework honesty: Model- and framework-specific fields cross unchanged and remain visible in effective records instead of being translated into a common InferLab profile.
  4. Bounded surface: The endpoint contract names only the two demonstrated OpenAI routes and does not introduce an arbitrary route registry or Responses API.

The public definition field is request_body. A request fragment cannot replace fields owned by the resolved task or runtime. Invocation overrides target the same nested definition fields. A selection that does not resolve to exactly one individual task is rejected before inference; recipes are the composition boundary for multiple tasks.

  • Estonia and other generative tasks can record and vary real temperature, reasoning_effort, and chat_template_kwargs values.
  • Likelihood evaluation keeps its established completions prompt-logprob probe and scoring path.
  • Each Eval has one task, one request family, one route, and one request fragment; recipes provide explicit multi-task composition.
  • Dry-run, execution, and records can derive from one resolved request without reconstructing configuration.
  • Backend capability claims remain tied to the exact route and topology exercised.
  • The adapter endpoint contract and generated projections must change to carry two named paths.
  • InferLab-owned SGLang and TensorRT-LLM proxies need substantive chat handoff support rather than a route alias.
  • A chat-capable framework can still reject a request for a model without a usable chat template; qualification therefore remains model- and route-specific.
  • Operators cannot pass a selection that expands to multiple lm-eval tasks as one Eval definition; they express the intended task set explicitly in a recipe.
  • Framework extensions remain backend-owned and can differ even when their JSON field names match.
  • Completions remains part of the product contract rather than becoming a deprecated compatibility route.

Select the OpenAI request form from measurement semantics and preserve a definition-owned request-body fragment (accepted)

Section titled “Select the OpenAI request form from measurement semantics and preserve a definition-owned request-body fragment (accepted)”
  • Pros: Generative tasks use the model’s real chat template and reasoning controls, Likelihood scoring retains completions prompt-logprob semantics, Definitions and records preserve exact request fields without cross-framework translation
  • Cons: The endpoint contract and built-in proxies must support two named routes

Use chat-completions for every request (rejected)

Section titled “Use chat-completions for every request (rejected)”
  • Pros: One generation endpoint for most modern models
  • Cons: Cannot reproduce completions echo and prompt-logprob scoring, Makes the lowest-level smoke depend on a usable chat template
  • Rejected because: A single chat route cannot preserve the established likelihood and smoke workflows

Let each Eval or Bench definition select completions or chat-completions (rejected)

Section titled “Let each Eval or Bench definition select completions or chat-completions (rejected)”
  • Pros: Maximum operator control
  • Cons: Allows a request form that contradicts the resolved lm-eval output type, Creates invalid combinations and a second task-semantics authority
  • Rejected because: Request form follows task semantics and is not an independent experiment parameter

Normalize sampling and reasoning controls into an InferLab inference profile (rejected)

Section titled “Normalize sampling and reasoning controls into an InferLab inference profile (rejected)”
  • Pros: One apparent parameter vocabulary across frameworks
  • Cons: Requires lossy and version-sensitive translation, Hides the exact request that a backend received
  • Rejected because: InferLab should preserve real request fields rather than own a drifting framework abstraction