Skip to content

Decision record · ADR

ADR-0040: Replay serving-Bench populations from operator-curated files

Status: accepted | Date: 2026-08-22

References: RFC-0004:C-BENCH-REQUEST-SOURCES, RFC-0004:C-BENCH-CACHE-STATE, RFC-0005:C-BENCH-REQUEST-SOURCE-EVIDENCE, ADR-0034

Downstream comparator work needs to replay an already-materialized Bench request population — paired runs against external benchmark harnesses, outlier reproduction, and controlled A/B reruns on the Specialized Engine workspace — with the exact same prompts, possibly subset, reordered, or edited by the operator. The serving-Bench request sources are a closed set under RFC-0004:C-BENCH-REQUEST-SOURCES: random and random_mixture derive their populations from the Bench seed, dataset resolves release-catalog snapshots, and the catalog rule explicitly forbids definitions from supplying an arbitrary local path or loader. Operators work around this today by freezing population files beside the schema and hand-aligning digests with external scripts.

Bench records already retain the materialized population as a record artifact (cases/request-source/artifacts/population.jsonl), so InferLab-produced populations are addressable local files; the record digests the population under RFC-0005:C-BENCH-REQUEST-SOURCE-EVIDENCE, and the release-owned Bench runner is the population preparation authority. The decision drivers are: strict replay must rest on content-derived identity rather than on a locator; the replayed population must compose with prefix-sharing geometry and cache-state conditioning under RFC-0004:C-BENCH-CACHE-STATE like any other prompt provider; and InferLab should not grow a selection or filtering language that operator tooling already covers.

Add a replay request-source kind whose definition names a workspace-relative population file in the InferLab materialized population format, an optional expected SHA-256 digest, and the prompt kind. The file content is the sole population authority: entry order, subset selection, duplication, and editing are expressed by the file itself, not by an InferLab selection layer. The Bench runner validates the declared digest and each entry’s shape during population preparation and digests the file into the record, so downstream of preparation a replayed population is indistinguishable from a generated one: warmup and profiling partitioning, prefix-sharing declarations, and primed or cold cache-state conditioning apply unchanged. No export facility is added: records already retain the materialized population artifact at a documented path, and operators copy it into the workspace and pin its digest when they want to replay a previous run.

Positive: strict replay of generated, frozen, or edited populations becomes a declared, digest-pinned Bench source instead of an out-of-schema convention; paired comparator runs against external harnesses re-import a frozen population inside the schema; primed and cold replay work through the existing cache-state path with no special cases; no population export command is needed because records already retain the materialized artifact.

Negative: the definition references operator-managed local state, so record portability stops at the digest — a record proves which content ran, not where to fetch it, and a replay definition fails at preparation once its file is gone; operators must copy a population out of a record before that record is cleaned; the closed-source contract gains one deliberately scoped exception that must not become precedent for unmanaged inputs.

Neutral: the Bench runner remains the population authority and additionally owns replay validation; AIPerf consumes the replayed population through the same frozen-population boundary as other sources; subset and transformation tooling stays outside InferLab in ordinary operator scripts.

Operator-curated population file as the sole authority (accepted)

Section titled “Operator-curated population file as the sole authority (accepted)”
  • Pros: Single content-addressed authority: the digest is the identity and the path is only a locator, No derivation layer to specify, test, or carry in record evidence, Arbitrary subsets, orders, duplication, and edits stay in ordinary operator tooling, Replay is a pure prompt provider, so primed and cold cache states and prefix-sharing geometry compose without special cases
  • Cons: Definitions reference operator-managed local files whose loss makes them unplannable, mitigated by digest pinning and typed preparation failure, Requires an explicit RFC carve-out from the closed-catalog no-local-path rule

InferLab-owned selection layer (offset/stride/count or ordered spans) over a source population (rejected)

Section titled “InferLab-owned selection layer (offset/stride/count or ordered spans) over a source population (rejected)”
  • Pros: Records could prove the derivation from a named source population to the replayed subset
  • Cons: Half-expressive: content filtering and entry edits remain impossible without growing a general workflow language, Creates two-level provenance (source digest, selection declaration, derived digest, per-entry source indices) that every record must carry
  • Rejected because: Selection is already expressed more generally by curating the file with standard tools; an in-schema selection layer duplicates operator tooling while remaining less capable.

Extend the dataset catalog with workspace-local entries (rejected)

Section titled “Extend the dataset catalog with workspace-local entries (rejected)”
  • Pros: Reuses the acquisition, digest, and cache machinery, Keeps a named dataset indirection between definition and file
  • Cons: Catalog entries carry release-owned materialization semantics (structured messages, held-out targets, tail rollback) that do not map to an already-materialized population, Splits source registration and subset selection across two declaration sites
  • Rejected because: A replayed population is materialized by definition; routing it through dataset materialization adds an authority layer without adding an invariant.

Record-referenced replay only (no operator files) (rejected)

Section titled “Record-referenced replay only (no operator files) (rejected)”
  • Pros: Closed world: replay sources come only from InferLab’s own records
  • Cons: Cannot import externally frozen populations, which is the paired-comparator workflow that motivated the feature
  • Rejected because: Too narrow for the demonstrated workflow; a record reference can later be added as sugar that resolves to the same file-plus-digest authority.