Skip to content

Decision record · ADR

ADR-0017: Model serving Bench inputs as closed request sources

Status: accepted | Date: 2026-07-16

References: RFC-0003, RFC-0004, RFC-0005, ADR-0016

Serving Bench currently stores random prompt shape as flat input_tokens and output_tokens fields on both static and adaptive definitions. Adding a release-qualified conversational dataset creates different input and output semantics: random input length is exact, dataset input length is a maximum applied to message history, random output length is required, and dataset output length may derive from a held-out assistant target.

InferLab needs one public Bench definition shape that excludes invalid combinations, keeps load and SLO policy independent from request origin, and lowers every supported origin to the confirmed one-sample-one-request execution model. The change occurs before the new dataset surface is released, so one coordinated recipe migration is cheaper than retaining two authorities.

RFC-0003 requires typed invocation overrides and one frozen resolved execution. RFC-0004 keeps serving Bench semantics framework-neutral and execution in the release-pinned AIPerf runtime. RFC-0005 requires effective request and native artifact evidence. ADR-0016 selects chat-completions for serving Bench from measurement semantics. InferLab does not expose arbitrary dataset URLs, formats, AIPerf dataset types, or a general workload language.

We will replace flat Bench token-shape fields with one closed request_source union because:

  1. Semantic precision: the random variant owns exact input and output lengths, while the catalog-dataset variant owns a dataset identity, maximum input length, and optional fixed output override.
  2. One execution boundary: both variants resolve to a deterministic population in which one entry produces one transport request; load and measurement policy consume that population without knowing its origin.
  3. Bounded extensibility: a new request origin requires another demonstrated closed variant, not arbitrary paths, formats, operations, or a new Bench kind.
  4. One authority: the unreleased flat shape is migrated once and is not retained as a compatibility alias.

The Bench-level seed remains the authority for source randomness and existing stochastic load behavior. Invocation overrides may change fields within the selected variant but do not switch its kind. Dataset names resolve only through the release-qualified catalog. The resolved request population, rather than source-specific configuration, is the authority consumed by dry-run when available, execution, and records.

  • Workspace definitions state exact-length generation and bounded conversational sampling without overloaded field names.
  • Static, adaptive, warmup, normalization, and SLO code can remain source-independent after resolution.
  • AIPerf receives a frozen request population and does not own catalog selection, network fetching, or resampling.
  • Future catalog datasets can reuse the same dataset variant when they satisfy the same single-request workflow.
  • Every current random Bench definition and cross-language protocol projection changes together; mitigation is one repository-wide migration with no compatibility branch.
  • Dataset-derived output lengths require resolved distribution evidence instead of one definition scalar; mitigation is to freeze the population before cases and validate metric applicability from its output policy.
  • Nested override paths become longer; mitigation is that the existing TOML-path override mechanism already traverses typed tables.
  • The catalog and source cache are InferLab release/control-plane concerns, while tokenizer-dependent conversation materialization remains in the versioned Bench toolchain.
  • Concrete catalog membership remains release-owned evidence rather than an enumeration in the architecture decision.

Use a closed request_source union with random and catalog-dataset variants (accepted)

Section titled “Use a closed request_source union with random and catalog-dataset variants (accepted)”
  • Pros: Variant-specific fields make invalid random/dataset combinations unrepresentable, Static and adaptive load, SLO, warmup, timeout, and request_body policy remain shared, Both variants resolve to one request-population boundary
  • Cons: Existing random definitions and protocol projections require one coordinated migration

Add flat optional dataset and max-input fields beside the existing token fields (rejected)

Section titled “Add flat optional dataset and max-input fields beside the existing token fields (rejected)”
  • Pros: Smallest textual change to existing random definitions
  • Cons: Validation must interpret several conditional field combinations, input_tokens and output_tokens acquire source-dependent meanings
  • Rejected because: The apparently small migration leaves ambiguous states and two shape vocabularies in the public API

Create separate serving kinds for random and dataset workloads (rejected)

Section titled “Create separate serving kinds for random and dataset workloads (rejected)”
  • Pros: Each kind can have an independent field set
  • Cons: Duplicates static and adaptive load, SLO, warmup, timeout, and request-body policy, Future request origins multiply measurement kinds
  • Rejected because: Request origin does not own execution or load semantics and does not justify a new measurement kind